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Project  Summary 

This  report  documents  a  Phase  I  STTR  effort  to  develop  a  prototype  Agent-Enhanced 
Electronic  Classroom  (eClass)  that  uses  intelligent  agents  as  tutors  and  coaches.  In  this  Phase  I 
project,  we  worked  alongside  the  UMass  Dartmouth  Division  of  Continuing  Education  to 
investigate  an  electronic  classroom  architecture  which  will  complement  the  training  objectives  of 
the  Five  Pillars  of  the  WF  XXI  Campaign.  The  architecture  developed  incorporates  intelligent 
software  agent  technology  capable  of  personalizing  course  content  to  each  student. 

The  eClass  prototype  was  completed  during  this  Phase  I  effort,  and  is  available  through  the 
web  at  http://agents.cra.com/eclass  .  The  prototype  requires  a  Netscape  3.0  (or  higher)  browser 
running  on  Windows  95/NT  operating  systems.  The  selected  training  content  for  the  prototype 
involves  learning  how  to  interpret  aviation  weather  reports.  The  eClass  Agent  allows  students  to 
navigate  through  course  content  (a  coded  weather  report),  bringing  the  student  quizzes  and 
exams  when  the  student  has  reviewed  the  corresponding  material.  The  agent  corrects  the  exams 
and  coaches  the  student  through  additional  material.  The  eClass  architecture  also  provides  links 
to  external  resources,  a  question  &  answer  agent  that  communicates  with  an  instructor,  and  a 
student  notepad  for  personal  course  notes. 

Evaluation  of  the  prototype  eClass  was  conducted  primarily  at  the  Eastern  Aviation  Army 
National  Guard  Training  Site  in  Ft.  Indiantown  Gap,  Pennsylvania.  Two  sets  of  students  were 
given  access  to  the  web-based  weather  eClass  as  a  supplement  to  their  in-class  aviation  training. 
Feedback  from  the  first  group  of  students  was  used  to  improve  the  eClass  interface  and  delivery 
mechanisms,  enabling  the  second  group  of  students  to  successfully  utilize  the  eClass  weather 
training  to  improve  their  knowledge  and  performance. 

This  Phase  I  effort  has  produced  two  significant  prototypes  with  high  commercial  potential. 
The  first  is  a  general  agent-enabled  electronic  classroom  framework  that  can  easily  be 
customized  for  any  number  of  web-based  training  materials,  including  interactive  Java 
simulations.  The  second  potential  product  is  a  Multimedia  Question  &  Answer  Agent  that  can  be 
used  within  any  eClass,  or  as  a  stand-alone  interactive  agent,  to  provide  direct  and  immediate 
answers  to  questions  posed  over  the  web  (http://agents.cra.com/q&a  ).  The  prototyped  Q&A 
Agent  is  easily  customizable  through  a  Q&A  Administration  agent,  to  be  used  as  web  FAQ 
administration  agents,  tech  support,  or  interactive  help  desks.  Both  the  eClass  and  Q&A  Agents 
are  currently  being  evaluated  in  commercial  web  sites  and  on-line  university  courses. 
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1.  Introduction 

Training,  according  to  the  Chief  of  Staff  of  the  Army,  is  “the  cornerstone  of  readiness”  and 
as  such  “remains  the  Army’s  most  important  peacetime  mission”  (Lengyel,  1996).  In  particular, 
the  Army  needs  cost-effective  and  efficient  training  techniques  and  materials,  given  the 
increasingly  sophisticated  and  informed  threats  in  today’s  worldwide  battlefield.  WarFighter 
XXI  (Lengyel,  1996)  describes  the  Army’s  vision  for  training  in  the  21st  century.  The  Army 
Long  Range  Training  Plan  of  WF  XXI  outlines  a  plan  to  implement  a  distance  learning  program 
by  moving  portions  of  resident  training  from  classrooms  to  soldiers  at  home  station.  Some  of  the 
goals  of  the  Army’s  distance  education  program  (Lengyel,  1996)  are  to: 

•  train  Commanders  in  military  tactics  and  procedures,  the  art  of  leadership,  and  map  reading 
and  visualization, 

•  train  both  active  and  reserve  warfighters  to  survive  on  both  the  physical  and  the  digital 
battlefield, 

•  use  advanced  technologies  to  teach  more  students  with  fewer  instructors, 

•  lower  costs  through  decreased  travel  and  housing  costs,  and 

•  automate  course  production  and  eliminate  redundant  courses. 

The  Standard  Army  Training  System  (SATS)  and  Training  Support  Package  (TSP)  are  two 
of  the  five  components  of  WF  XXL  These  programs  include  tools  for  providing  instructors  with 
training  templates,  matrices  and  aid  in  the  management  of  training  resources.  The  stakeholders  in 
the  distance  education  program  include  the  instructional  designers  (i.e.,  those  involved  in 
designing  SATS  and  TSP),  the  unit  trainers  (i.e.,  those  responsible  for  selecting  training 
material),  and  the  students  (commanders  and  soldiers). 


Table  !•!:  Internet  Enabling  Technologies 


Type 

Technologies 

Content  presentation 

Hypertext,  Shockwave,  quicktime  video,  Java, 
VRML,  Real-Audio,  CU-SeeMe 

Communication 

Email,  listsrv,  MUDs,  MOOs,  newsgroups,  chat 
rooms,  video  conferencing,  Look@Me 

Knowledge  discovery  and 
off-line  automation 

Metalists,  search  engines,  virtual  libraries,  online 
news,  USENet,  softbots,  agents 

Electronic  classrooms 

WEST,  HyperCourseware,  CyberEd 

Internet  technologies  including  those  listed  in  table  1-1  have  seen  an  explosion  in  growth  and 
usage  over  the  past  several  years  (Witten,  Moffat  &  Bell,  1994).  The  introduction  of  the  internet 
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and  related  technologies  offers  an  opportunity  to  expand  the  potential  of  computerized  training 
by  removing  the  physical  boundaries  defined  by  traditional  training  techniques:  Internet-based  or 
web-based  training  (WBT)  offers  a  venue  for  students  to  learn  in  a  virtual  or  electronic 
classroom  by  exploring  and  learning  from  centrally  stored  curriculum  materials,  and 
collaborating  and  communicating  with  instructors  and  peers  via  email  and  online  forums. 

Numerous  examples  of  electronic  classrooms  have  appeared  in  recent  years. 
HyperCourseware  (Norman,  1994)  is  a  prototype  system  for  the  preparation  and  presentation  of 
materials  in  an  electronic  teaching  environment.  It  consists  of  a  Home  Screen  and  a  set  of 
interconnected  modules  organized  into  three  groups:  Course  Materials,  Course  Products,  and 
Course  Tools.  HyperCourseware  runs  in  either  the  Windows  or  Macintosh  environments.  WEST 
(WEST,  1996)  is  an  example  of  a  commercial  virtual  classroom  product  that  provides  remote 
authoring  and  administration  facilities  to  create  course  content  and  manage  students.  WEST  runs 
over  networks  such  as  the  Internet  and  corporate  networks  and  it  features  online  course  pages, 
built-in  email  and  discussion  forums  and  electronic  homework  submission  capabilities.  CyberEd 
(Stone,  1996)  is  a  selection  of  standard,  fiill-credit  University  courses  being  offered  to  the  global 
audience  of  the  Web  through  the  UMass  Dartmouth  Division  of  Continuing  Education.  CyberEd 
uses  the  World  Wide  Web,  email,  and  other  Internet  resources,  to  provide  opportunities  for 
meaningful  student-to-faculty  and  student-to-student  interaction. 

In  this  Phase  I  effort,  Charles  River  Analytics  worked  alongside  the  UMass  Dartmouth 
Division  of  Continuing  Education  to  investigate  an  “agent-enhanced  electronic  classroom” 
architecture  which  complement  the  training  objectives  of  the  Five  Pillars  of  the  WF  XXI 
Campaign.  The  architecture  expands  on  the  CyberEd  concept  by  incorporating  intelligent 
software  agent  technology  capable  of  personalizing  course  content  to  each  student.  Intelligent 
software  agents  are  self  contained  software  programs  that  operate  in  computer  environments 
such  as  operating  systems,  databases,  and  applications.  The  combination  of  knowledge, 
inference,  and  communication  components  of  an  agent  results  in  behavior  that  is  intelligent  in 
the  sense  that  the  agent  has  sufficient  expertise  and  ability  to  complete  a  task  autonomously  with 
little  or  no  human  intervention. 

Table  1-2  summarizes  the  features,  advantages,  and  benefits  of  our  proposed  agent-enhanced 
electronic  classroom  architecture.  Agents  and  electronic  classrooms  offer  the  Army  a  versatile 
training  tool  and  medium  which  will  benefit  the  Army  as  a  whole  as  well  as  each  of  the  distance 
education  stakeholders  (designers,  trainers  and  students).  We  believe  that  this  tool  has  potential 
in  the  Army,  government  and  commercial  sectors. 
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Table  1-2:  Features,  Advantages,  and  Beneflts  of  agent-enhanced  electronic  classroom 


xr"^:  1  '  \ 


Extensible  and  flexible 
architecture 


Can  be  centralized,  decentralized, 
networked  or  standalone 


Customize  architecture  and  security 
to  suit  needs 


Platform  independent 
software 


Runs  on  mainstream  hardware  (PCs,  Macs, 
workstations)  _ 


No  need  to  purchase  new  hardware 
and  software 


Centralized  course  content 
repository 


Course  content  can  be  updated 
Immediately 


Reduced  production  and  distribution 
costs  and  time 


Flexible  course  participation 


Students  can  view  material  at  any  time  of 
day,  in  familiar  surroundings,  and  at  any 
rate 


Students  learn  at  their  own  pace 


Intelligent  agents 


Learns  user’s  current  and  persistent 
information  interests 


Personalizes  web  training  content, 
promotes  discovery  learning,  alerts 
trainer  to  student  weaknesses 


1.1  Phase  I  Technical  Objectives 

The  objective  of  this  Phase  I  effort  was  to  assess  the  feasibility  of  developing  an  architecture 
for  an  agent-enhanced  electronic  classroom  that  incorporates  agent  technology.  Basic  questions 
addressed  during  the  effort  were: 

•  How  can  we  develop  an  architecture  for  an  electronic  classroom  that  is  flexible  and 
extensible? 

•  How  can  the  architecture  be  prototyped  using  the  CyberEd  concept?  What  kinds  of  hardware 
and  software  are  necessary  to  build  an  electronic  environment?  What  are  security  provisions? 

•  How  can  the  architecture  be  enhanced  with  agent  technology?  In  particular,  how  can 
intelligence  be  incorporated  into  the  architecture?  What  kinds  of  intelligent  advice  can  it 
offer?  What  are  the  ramifications  of  embedded  intelligence  on  the  instructional  designer 
and/or  trainer? 

•  In  general,  what  are  the  features  of  courses  that  can  be  successfully  taught  via  an  electronic 
classroom?  Can  we  use  a  local  ROTC  program  to  prototype  an  Army  course  to  demonstrate 
feasibility? 

•  How  will  such  an  architecture  benefit  the  Army?  How  can  the  five  Pillars  of  WF  XXI 
benefit?  How  will  the  distance  education  stakeholders  (designers,  trainers  and  warfighters) 
benefit? 

•  What  criteria  will  be  used  to  evaluate  the  agent-enhanced  electronic  classroom  prototype? 
How  can  evaluation  be  conducted? 

The  Phase  I  effort  addressed  these  questions  and  demonstrated  feasibility  of  the  approach 
through  on-line  prototypes  and  student  evaluations.  As  a  result  of  this  effort,  we  are  in  a  position 
to  outline  a  Phase  II  program  of  full-scope  prototype  design  and  development. 
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1.2  Summary  of  Phase  I  Results 

Our  Phase  I  study  centered  on  the  development  and  feasibility  assessment  of  an  agent- 
enhanced  electronic  classroom  for  the  web.  Six  development  tasks  comprised  our  effort: 

•  Identification  of  State-Of-The-Art  in  Electronic  Classrooms 

•  Determination  of  Prototype  Lesson  Domain 

•  System  Architecture  Design 

•  Implementation  of  a  Limited-Scope  Prototype 

•  Demonstration  and  Evaluation  of  Concept  Prototype 

•  Requirements  Specification  and  Commercialization 

We  first  defined  the  Phase  I  problem  scope.  This  involved  consultation  with  the  sponsor, 
through  which  we  selected  the  interpretation  of  aviation  weather  reports  as  the  lesson  content 
for  the  prototype  electronic  classroom.  The  following  requirements  for  the  Phase  I  effort  were 
identified: 

•  The  lesson  content  must  have  inherent  structure  that  can  be  navigated  by  students. 

•  The  lesson  must  be  testable  through  multiple  choice  questions 

•  The  tutor  agent  must  monitor  student  progress  through  content 

•  The  tutor  agent  must  administer  exams  and  provide  coaching 

•  The  electronic  classroom  must  demonstrate  multimedia  capabilities 

•  The  electronic  classroom  must  be  extensible  to  other  lesson  content 

•  The  agent  tutor  must  facilitate  student-instructor  interaction  and  communication 

With  the  problem  scope  and  requirements  identified,  the  target  lesson  content  was  selected  to 
be  a  limited  subset  of  the  Army  National  Guard  Aviation  Training  course.  The  selected  subset  of 
aviation  training  was  Interpreting  Aviation  Weather  Reports.  This  is  a  component  of  the  MOS 
93P  Aviation  Operations  Specialist  Course  that  mold  to  the  METAR  format.  This  content 
provided  a  complete  lesson  that  could  be  presented  in  its  entirety,  while  remaining  small  enough 
to  complete  in  the  Phase  I  timeframe. 

The  agent-enhanced  electronic  classroom  (eClass)  architecture  was  chosen  such  that  it  would 
be  easily  extensible  for  additional  in-depth  content  and  to  other  lesson  domains.  To  accomplish 
this,  a  Java-based  client-side  implementation  was  used  along  with  Cold  Fusion  server-side 
connections  to  a  MS  Access  database.  All  relevant  lesson  content  information  is  stored  within 
MS  Access  tables  and  HTML  files,  while  the  Java  code  contains  the  agent  tutor  rules  and  general 
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eClass  functionality.  In  this  manner,  lesson  content  can  be  easily  modified  and  replaced  without 
changing  the  basic  functionality  of  the  eClass  and  its  agent  tutor. 

The  eClass  architecture  was  prototyped  for  the  aviation  weather  report  training,  and  is 
available  at  http://agents.cra.coni/eclass  for  demonstration  purposes.  The  results  obtained  from 
this  Phase  I  prototype  effort  can  be  summarized  as  follows: 

•  a  three-frame  approach  to  the  eClass  user  interface  provides  a  consistent  learning 
environment  in  which  there  are  designated  areas  for  student  self-paced  navigation, 
agent  tutoring  interaction,  and  lesson  content  presentation. 

•  the  eClass  client-server  architecture  is  capable  of  accommodating  a  large  range  of 
training  content,  providing  easy  customization  for  developers  and  simplified 
maintenance  for  instructors. 

•  the  Agent  tutor  is  effective  in  monitoring  student  navigation  through  course  content, 
capable  of  administering  exams  when  needed,  successfully  corrects  exams,  and 
provides  accurate  coaching  for  areas  the  student  had  difficulty  with. 

•  the  Question  &  Answer  Agent  component,  of  the  prototyped  eClass  successfully 
provides  direct  and  accurate  answers  to  student  questions,  decreasing  the  load  on 
actual  course  instructors. 

•  the  Question  &  Answer  Agent  component  is  capable  of  serving  as  a  stand-alone 
service,  and  was  demonstrated  for  the  commercial  OpenSesame  web  site,  Charles 
River  Analytics  intranet,  and  UMass  Dartmouth  CyberEd  course. 

•  the  prototyped  eClass  for  Weather  training  was  evaluated  by  two  groups  of  Army 
National  Guard  students.  Feedback  from  the  first  group  of  students  was  used  to 
modify  the  agent  functionality  for  the  second  group.  The  second  group  of  students 
successfully  completed  the  eClass  training  using  the  on-line  prototype. 

Based  on  the  results  of  our  Phase  I  effort,  the  agent-enhance  eClass  architecture  has 
significant  commercial  potential  for  military,  university,  and  corporate  training  applications. 


1.3  Report  Outline 

This  report  summarizes  and  documents  our  Phase  I  STTR  effort  to  develop  and  evaluate  a 
prototype  agent-enhanced  electronic  classroom  for  the  internet  environment.  Chapter  2  presents  a 
background  on  the  web-based  training  domain  in  addition  to  research  conducted  during  Phase  I. 
Chapter  3  discusses  the  training  pedagogy  and  introduced  the  selected  training  content  for  the 
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developed  prototype.  Chapter  5  presents  the  detailed  eClass  architecture  and  prototype 
implementation.  The  prototype  evaluation  is  summarized  in  Chapter  6  with  detailed  results 
shown  in  Appendix  A.  The  Phase  n  requirements  and  commercialization  concepts  are  presented 
in  Chapter  7  with  final  project  summary  and  conclusions  in  Chapter  8. 
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2.  Background 

This  chapter  provides  technical  background  on  past  research  and  current  technologies  most 
relevant  to  our  effort  to  develop  an  agent-enhanced  electronic  classroom. 

2.1  Web-based  Training  Evolves  from  Computer-based  Training 

Over  the  years,  computer-based  training  (CBT)  has  evolved  from  linear  and  text-based 
systems  into  user-centered,  instructional  systems  that  utilize  graphics,  photographs,  animation, 
sound  and  motion  video.  Such  software  is  often  called  multimedia.  Authoring  tools  such  as 
Director,  Visual  Basic,  SuperCard  and  HyperCard  allow  developers  to  focus  more  on  the 
instructional  benefits  of  multimedia  than  on  the  technical  hurdles  required  to  build  them.  The 
commercial,  industrial,  military  and  government  sectors  recognize  the  effectiveness  and  cost 
benefits  of  well-designed  CBT,  which  are  enhanced  by  self-paced  and  any  time/place  delivery.  A 
study  revealed  that  students  retain  70%  of  information  taught  through  CBT  and  only  10%  taught 
through  lecture  (Lager,  1994). 

A  major  weakness  of  CBT  is  that  applications  require  software  and  hardware.  CBT 
applications  are  susceptible  to  changes  in  both  the  operating  system  platform  and  content  and 
applications  are  difficult  to  economically  and  quickly  update.  CBT  can  also  be  difficult  or 
expensive  to  port  to  other  platforms. 

Web-based  training  represents  an  evolution  of  the  concept  of  CBT.  It  also  presents  solutions 
to  the  many  problems  posed  by  CBT.  The  advantages  and  benefits  of  WBT  include: 

•  Applications  are  usually  written  in  a  standard  language  (HTML),  an  open  standard. 

•  Applications  run  on  multiple  platforms  (Windows,  Mac,  UNIX). 

•  Content  can  be  easily  and  remotely  updated. 

•  The  course  author  can  link  to  other  materials  drawing  on  the  global  resources  of  the  Web  to 
enrich  the  lessons. 

•  Content  can  be  viewed  at  any  time  and  rate  and  in  familiar  surroundings  (this  also  obviates 
the  need  to  schedule  classes,  instructors  and  classrooms). 

•  Software  for  presenting  the  course  is  inexpensive  (both  browsers  and  servers  are  available  for 
free  on  the  Internet). 

In  addition  to  being  used  to  publish  course  content,  WBT  can  also  incorporate  technology 
which  will  provide  the  capability  for  communication  and  collaboration  among  students  and  their 
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peers  and  between  students  and  their  professors.  Testing  students  via  the  Internet  to  verify  they 
have  assimilated  information  is  also  possible  with  this  technology. 

2.2  State-Of-The-Art  Research 

A  study  of  current  research  in  the  area  of  CBT  and  WBT  has  identified  the  following  centers 
and  organizations  of  interest: 

The  Web-Based  Training  Information  Center  is  a  non-profit  resource  for  individuals  and 
organizations  interested  in  developing  and  delivering  training  using  Web  technology.  The  intent 
of  this  site  is  to  share  non-proprietary  information,  stimulate  creative  ideas,  and  link  to 
interesting  training  sites  around  the  world. 

The  m  Online  Training  Center  is  a  collection  of  on-line  courses  designed  to  help  you 
quickly  master  a  variety  of  computer-related  topics. 

Prentice  Hall  is  delivering  Online  Applications  to  supplement  a  number  of  their  books. 

The  IT  (Information  Technology)  Works  CD-ROM  represents  a  new  generation  in 
college-level  courseware.  IT  Works  is  an  innovative  multimedia  educational  tool  that  can  work 
one-on-one  with  students  to  interactively  demonstrate  many  important  computer  concepts  and 
applications.  This  extremely  visual  and  interactive  courseware  employs  sound,  motion  video, 
colorful  high-resolution  graphics,  and  animation. 

PHLIP  is  a  content-rich,  multi-disciplinary  business  Web  site  created  by  professors  for 
professors  and  their  students.  With  PHLIP,  faculty  and  student  support  moves  into  the 
information  age.  It  is  intended  to  create  a  web  based  "Learning  Environment"  that  will  consist  of 
numerous  links  to  websites,  a  current  events  reporting  feature,  online  chat,  and  bulletin  board 
functionality.  It  is  fully  prototyped,  and  will  evolve  to  reference  a  considerable  amount  of 
text/media  assets  from  our  Bus  F*ublishing  archive. 

MetaKnowledge  is  developing  new  software  technology  for  interactive  learning  via  the 
internet.  The  company  is  currently  developing  its  first  product  named  MetaKnowledge  Builder. 
This  new  technology  will  enable  the  rapid  production  and  management  of  high  quality 
knowledge  based  online  content.  The  company  was  founded  by  Rick  Spitz,  previously  Vice 
President  of  Software  at  Apple  Computer  in  Cupertino,  California. 

Engines  for  Education  ASK  systems  are  a  form  of  hypermedia  based  on  the  metaphor  of 
having  a  conversation  with  an  expert  (or  a  group  of  experts).  In  this  conversation,  the  user 
provides  questions  and  the  ASK  system  provides  the  answers.  In  a  real  conversation,  both 
participants  influence  the  flow  of  discussion.  In  an  ASK  system,  the  same  holds  true.  The  user 
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influences  the  flow  by  selecting  which  questions  to  pursue  and  the  ASK  system  influences  the 
flow  through  the  answers  it  provides. 

2.3  Distance  Learning  Projects 

Although  formal  standards  do  not  exist  for  creating  electronic  classrooms,  numerous  training 
sites  are  paving  the  way  by  offering  examples  of  how  courses  can  be  taught  online.  A  complete 
education  environment,  according  to  Norman,  1994,  requires  a  pedagogical  structure, 
interactivity,  and  hypermedia. 

Hyper Coursware  (Norman,  1994)  uses  the  conventional  objects  of  classroom  instruction 
and  implements  them  in  electronic  form  in  the  electronic  classroom.  ^  Objects  such  as  the  course 
syllabus,  the  lesson  plan,  the  lecture  notes,  the  class  roll,  etc.  are  instantiated  in  graphic  form  in  a 
hypermedia  database.  Furthermore,  in  HyperCourseware  the  hypermedia  database  is  used  to 
provide  the  same  sort  of  natural  links  between  objects  as  one  would  expect  in  the  educational 
materials  themselves.  For  example,  the  syllabus  is  a  natural  navigational  mechanism  to  jump  to 
lectures,  readings,  and  assignments;  the  classroll  is  a  natural  navigational  jump  to  information 
about  students  and  grades;  and  the  grade  list  is  a  natural  navigational  jump  to  exams  and 
assignments.  Each  of  these  are  syntactic  navigational  links  in  support  of  education. 

CyberEd  (Stone,  1996)  is  a  selection  of  standard,  full-credit  University  courses  being 
offered  to  the  global  audience  of  the  Web  through  the  UMass  Dartmouth  Division  of  Continuing 
Education.^  The  objective  is  to  create  a  distance  learning  environment  that  rivals  the  traditional 
classroom  environment  in  both  the  quality  and  content  of  the  learning  experience.  Thirteen 
courses  are  currently  being  offered  to  students  over  two  semesters  (Fall  ‘95  -  Spring  ‘96)  with 
several  more  coming  up  in  the  summer.  Enrollment  is  limited  in  CyberEd-one  of  the  goals  of  the 
program  is  to  keep  the  student/instructor  ratio  to  a  size  where  professors  can  provide  the  same 
level  of  personal  interaction  as  in  the  traditional  classroom.  Another  cornerstone  of  the  program 
is  the  focus  on  communication:  CyberEd  uses  the  World  Wide  Web,  email,  and  other  Internet 
communication  resources  such  as  Chat,  listsrvs,  and  web-based  conferencing  systems  to  provide 
plenty  of  opportunities  for  meaningful  student-to-faculty  and  student-to-student  interaction.  The 
program,  like  HyperCourseware,  emphasizes  the  importance  of  communication  and 
collaboration  between  students  and  instructors. 


-See  URL;  http://www.lap.umd.edu/hcwFolder/hcwHome.htinl 
^See  URL;  http://www.umassd.edu/cybered/distlearninghome.html 
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Calvin  College-  Cyberspace  for  Beginners.  Course  Description:  IDIS  W29  was  an  1996 
Interim  (January  term)  course  offered  by  the  Department  of  Communication  Arts  and  Sciences. 
This  course  reflects  the  basic  structure  of  the  CyberEd  process.  Much  of  the  language  in  the 
description  resembles  that  used  in  CyberEd  promotional  materials.  "Use  the  Internet  to  research 
intensively  a  topic  of  the  student's  choice  and  write  a  hyperlinked  "paper"  based  entirely  on  Web 
sources." 

Economics  Department  -  University  of  Colorado.  Here's  a  process  that  although  somewhat 
narrow  in  scope  defines  the  basic  process  by  which  an  instructor  creates  content  for  his  or  her 
students:  "The  hypertext  nature  of  the  Web  allows  the  teacher  to  direct  students  not  only  to 
essential  material  but  also  to  other  relevant  materials  available  on  the  Web...  the  course 
designer/instructor  adds  links  to  more  tangential  course  elements  that  the  instructor  has  written 
or  links  to  other  Web  sites  which  contain  complementary  information... The  Web  can  be  used  in 
many  different  ways,  from  providing  a  simple  set  of  references  to  a  complete  compilation  of 
course  materials." 

The  World  Lecture  Hall  (http://www.utexas.edu/world/lecture/)  contains  links  to  pages 
created  by  faculty  worldwide  who  are  using  the  Web  to  deliver  class  materials.  For  example, 
you  will  find  course  syllabi,  assignments,  lecture  notes,  exams,  class  calendars,  multimedia 
textbooks,  etc. 

CASO's  Internet  University  (http://www.caso.com/index.html)  has  more  than  1,130  pages 
of  ONLINE  ED  information,  provided  to  the  Educational  and  Internet  communities  as  a  service 
of  CASO  (Cape  Software). 

2.4  Enabling  Internet  Technologies 

This  section  describes  most  of  the  enabling  internet  technologies  from  table  1-1,  and 
highlights  their  strengths  in  training  applications. 

2.4.1  Content  Presentation  Technologies 

Content  presentation  technologies  present  information  such  as  text,  video,  sound  and  images 
to  the  user.  Some  of  these  technologies  are  interactive,  requiring  a  response  from  the  user.  Other 
information  is  static,  containing  information  for  the  user  to  read,  listen  to  or  view.  Static 
information  is  usually  presented  in  a  linear  synchronous  format.  Examples  of  technologies  for 
presenting  content  are  described  in  table  2.4-1. 
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Table  2.4*1:  Content  Presentation  Technologies 


Technology 

Definition 

Best  Training  Use 

Hypertext 

Electronic  versions  of  a  printed  document.  Hypertext,  or 
hottext  is  text  that  is  linked  to  other  pages. 

Presenting  text 

Java 

Programming  Language  which  allows  applications  to  be 
run  on  any  platform  supporting  a  Java  enhanced  web 
browser. 

Custom-made  interactive 
applications 

VRML 

Virtual  3-D  graphical  scenes  which  the  user  can  move 
through  and  “experience”. 

Custom  made  interactive 
applications  which  require  third 
dimension 

Shockwave 

Software  which  allows  users  to  interact  with  Director 
presentations  (animations,  buttons,  URL  links,  videos)  in  a 
Netscape  Navigator  window. 

Custom  made  interactive  Director 
applications 

Real  Audio 

Provides  live  and  on-demand  real-time  audio  over  the 
Internet. 

Presenting  sound 

Quicklime  Video 

Provides  ability  to  view  movies  through  web  browser. 

Presenting  video  ciips 

CU-SeeMe 

Software  with  whiteboard  and  real-time  audio  which  can 
be  used  for  video  conferencing  from  workstation  to 
workstation  across  the  Internet. 

Whiteboard  and  real-time  audio 
could  be  used  to  create  content 
between  students  and  instructors. 

2.4.2  Communication  Technologies 

Technologies  which  allow  users  to  communicate  with  each  other  utilize  such  media  as  text, 
video,  bulletin  boards  and  sound.  Such  technologies  allow  for  collaboration  and  are  essential  for 
teams  to  collaborate  and  interact  over  a  distance.  Some  communication  technologies  are  defined 
in  table  2.4-2. 


Table  2.4-2:  Communication  Technologies 


Technology 

Definition 

Best  Training  Use 

Email 

Method  for  sending  correspondence  to  another  individual 
electronically. 

One-to-one  correspondence  which  is 
not  time  sensitive. 

Listsrv 

online  discussion  which  allows  participants  to  send 
identical  email  messages  to  all  the  members  of  a  list. 

One-to-many  correspondence  which 
is  not  time  sensitive. 

MUD,  MOO 

Text-based  and  graphics-based  virtual  environments  that 
provide  a  shared  space  and  activities  for  participants. 

Collaboration  and  communication 
among  two  or  more  individuals  in  real¬ 
time.  Good  medium  for  role  playing. 

Chatroom 

Text-based  environment  where  users  can  discuss  (by 
typing)  common  interests. 

One-to-one  or  one-to-many 
collaboration.  Good  medium  for 
meetings,  office  hours. 

Look®  Me 

Provides  ability  to  view  another  Look®  Me  user's  screen 
anywhere  in  the  world  in  real  time. 

Collaboration  on  documents  in  real 
time. 

CU-SeeMe 

Software  used  for  video  conferencing  from  workstation  to 
workstation  across  the  Internet. 

One-to-one  or  one-to-many 
communication  requiring  video  and 
sound  capabilities. 

Web  phone 

Internet-based  phone  systems  which  eliminate  long 
distance  calls. 

One-to-one  audio  communication. 
Could  be  used  in  conjunction  with 
Look®  Me. 
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2.4.3  Knowledge  Discovery  and  Off-line  Automation  Technologies 

Internet  technologies  which  promote  knowledge  discovery  give  users  the  power  and 
knowledge  to  explore  the  Internet  in  greater  depth.  Such  tools  support  the  discovery  learning 
theory  which  is  a  form  of  instruction  in  which  students  can  find  their  own  answers  to  questions. 
Technologies  which  do  not  require  user  to  be  present  to  conduct  work  via  the  Internet.  Some 
knowledge  discovery  and  off-line  automation  technologies  are  described  in  table  2.4-3. 


Table  2.4-3:  Knowledge  Discovery  and  Off-line  Automation  Technologies 


Technology 

Definition 

Best  Training  Use 

Search  engines 

Tools  which  search  the  World  Wide  Web  and 
produce  lists  of  sites  which  best  match  the  users 
search  criteria. 

Locating  resources  and 
references  in  real-time  or  while 
the  user  is  not  present. 

Online  news 

News  archive  sites. 

Reference  for  newsworthy 
events. 

USEnet 

Worldwide  network  of  information  bulletin 
boards  which  anyone  can  post  to. 

Reference  for  information 
pertaining  to  any  discussed  topic, 
and  forum  for  asking  questions. 

Softbots 

Intelligent  software  agents  which  are  given  a 
task  such  as  finding  information,  updating 
information  or  purchasing  information  on  the 
Internet  on  behalf  of  their  owner. 

Discovery  understanding  of  new 
information. 

User  Interface  Agents 

Intelligent  software  which  observes  the  user’s 
behavior  and  offers  to  automate  repetitive 
behavior  or  teach  concept.  i 

In-context  tutoring,  dynamic 
creation  of  content  and  tests 
based  on  user  interests. 

2.4.4  Open  Sesame  User  Interface  Agent 

Open  Sesame!  is  the  first  desktop  agent  that  learns  to  automate  the  routine  tasks  of  a  user. 
Open  Sesame!  has  been  reviewed  and  covered  in  trade  and  mainstream  publications  worldwide 
from  the  New  York  Times  to  the  Japanese  edition  of  PC  Week.  Open  Sesame!  has  been  shipping 
since  December  1993,  and  is  distributed  worldwide.  Currently,  we  are  shipping  version  1.1  of 
Open  Sesame!  which  is  specifically  optimized  for  Apple  Macintosh  computers  using  the 
PowerPC  RISC  chip.  (This  version  is  also  available  for  downloading  from  our  Web  page.) 

Here,  we  discuss  Open  Sesame!  as  an  example  of  a  software  learning  agent.  Open  Sesame! 
version  1.0  watches  for  two  kinds  of  tasks:  time-based  and  event-based  (Mazzu,  Snorrason, 
Caglayan  &  Wilkinson,  1994).  A  time-based  task  is  something  that  the  user  does  at  a  particular 
time.  For  example,  opening  electronic  mail  every  day  at  nine  o’clock  is  a  time-based  task.  An 
event-based  task  is  something  that  the  user  does  in  relation  to  another  task.  For  example,  opening 
the  clock  desk  accessory  before  logging  into  an  online  database  is  an  event-based  task. 

Open  Sesame’s  architecture  is  based  on  hybrid  neural  network/expert  system  technology 
(Mazzu,  Caglayan  &  Jonas,  1991)  and  is  shown  in  Figure  2.4-1. 
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Events 


inference 

a 

V 

Neural  Observations 

Learning  I 

instructions 

1 


Commands 


Figure  2.4*1  Open  Sesame!  Architecture 


Open  Sesame!  compares  the  high  level  events  (like  opening  a  folder,  quitting  an  application 
and  so  on)  generated  by  the  user's  mouse  clicks  and  keystrokes  to  information  stored  in  its  neural 
learning  module  and  in  its  inference  engine.  The  neural  learning  module  looks  for  repetitive 
patterns  that  haven’t  been  automated.  If  it  finds  one,  then  Open  Sesame!  creates  an  observation, 
such  as  the  one  shown  in  figure  2.4-2, 

Ohwfwtl—t 

Commandar,  1  noticu  you  often  4^ 

•  open  “Schedule’* 

•  offer  opening  “Project  Budget** 

mould  you  like  me  to  outomote  this  task? 

[  Edit...  )  f  NO  ] 

Figure  2.4*2:  Observation  Dialog  Box 


If  the  user  gives  her/his  approval,  then  Open  Sesame!  creates  an  instruction  to  automate  the 
task  that  it  observed.  The  inference  engine  compares  monitored  user  event  patterns  with  patterns 
for  existing  instructions.  When  it  finds  a  match.  Open  Sesame!  automatically  sends  a  set  of 
events  to  the  operating  system  to  perform  the  instruction. 

The  pattern  recognition  in  Open  Sesame!  uses  a  neural  network  paradigm  called  GEN- ART 
(Caglayan  &  Snorrason,  1993;  Snorrason  &  Caglayan,  1994),  based  on  Adaptive  Resonance 
Theory  (Carpenter,  Grossberg  &  Rosen,  1991)  to  categorize  the  high  level  events  it  monitors. 
One  advantage  of  ART  type  networks  is  their  provision  for  self-organized  learning,  which  does 
not  require  prior  knowledge  of  event-pattern  categories.  This  is  accomplished  via  bottom-up 
competitive  filtering  of  patterns  for  finding  the  “best  match”  category,  combined  with  top-down 
template  matching  for  determining  if  the  best  match  is  “good  enough.”  GEN-ART  allows  the  use 
of  customized  distance  metrics  to  handle  input  patterns  with  qualitative  (nominal  or  ordinal) 
variables. 
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3.  Pedagogy  and  Lesson  Content 

There  are  several  key  areas  in  which  the  work  for  this  contract  speaks  directly  to  pedagogical 
issues.  To  understand  the  benefits  of  this  work  in  this  regard,  it  is  important  to  understand  what 
educators  and  training  professionals  look  for  in  learning  technologies  —  what  they  hope  to  gain 
in  using  new  learning  technologies  in  the  classroom. 

Discussions  in  Usenet  news  and  on  educational  listserves  regarding  technology  in  the 
classroom  are  replete  with  controversy  and  conflict.  While  many  perspectives  exist,  for  the 
benefit  of  brevity,  this  report  will  focus  on  two  opposing  views. 

3.1  Pedagogical  overview 

There  is  a  model  of  distance  learning  that  says  communication  between  the  student  and 
instructor  does  not  need  to  take  place  at  all,  or  at  least  it  does  not  need  to  be  as  direct  or 
immediate  as  in  traditional  classrooms.  Proponents  of  this  perspective  support  the  self  paced,  self 
regulated  approach  to  instructional  design  and  favor  the  methods  set  forth  in  Computer  Based 
Training  (CBT). 

There  is  another  camp  that  propounds  a  model  of  distance  learning  in  which  the 
student/instructor  relationship  is  emphasized  and  technology  is  looked  to  as  a  means  to  better  the 
flow  of  communication  between  student  and  mentor.  Proponents  of  this  model  frown  upon 
automated  tasks  or  any  features  that  distance  the  instructor  from  interacting  with  students. 

It  is  the  opinion  of  this  author  that  a  synthesis  of  these  two  perspectives  is  the  key  to 
successful  development  of  agent  technology. 

3.2  Pedagogical  Advantages  of  Agents 

A  successful  implementation  of  agent  technology  in  today’s  marketplace  will  satisfy  the 
needs  and  expectations  of  all  pedagogical  assumptions  and  biases.  The  way  to  bridge  the  gaps 
between  parties  is  to  focus  on  what  technology  can  do  for  the  student. 

The  agent  has  the  ability  to  administer  quizzes,  freeing  instructors  from  the  task  of  deciding 
when  to  give  quizzes  as  well  as  the  actual  time  devoted  to  taking  them  in  class.  This  benefit 
could  easily  be  extended  and  made  more  elaborate  to  provide  completely  automated  courses  in 
which  the  instructor  is  entirely  absent  from  the  learning  experience.  But  what  is  more  toward  the 
point  of  developing  a  conunercially  and  educationally  viable  product  is  to  ask  if  an  automated 
course  is  in  the  student’s  best  interest  or  whether  it  would  be  better  to  include  the  instructor  but 
free  him  to  fulfill  other  roles. 
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In  a  keynote  address  given  to  the  UMass  Instructional  Technologies  Conference,  1997, 
Professor  James  Noblitt,  a  senior  fellow  at  the  Institute  for  Academic  Technologies  and  a 
member  of  the  Department  of  Romance  Language  at  the  University  of  North  Carolina,  Chapel 
Hill,  identified  three  key  areas  to  which  all  new  learning  technologies  should  make  a 
contribution.  He  related  that  new  technologies’  greatest  strengths  reside  in  the  issue  of 
authenticity  from  the  student  perspective. 

•  Authenticity  of  content:  Learning  technologies  should  expose  students  to  as 
much  primary  data  as  possible.  The  artifice  of  textbooks  often  removes 
students  from  the  actual  information  they  are  required  to  assimilate,  analyze, 
or  manipulate.  New  technologies  can  bring  primary  sources  of  information  to 
the  learning  experience  without  taking  the  learning  experience  out  of  the 
classroom. 

•  Authenticity  of  outcome:  Learning  technologies  should  assess  and  evaluate 
student  performance  and  skill  level  according  to  real  world  criteria.  New 
technologies  can  ensure  that  students  actually  understand  the  material 
presented  through  application  of  genuine  professional  criteria  either  through 
workplace  simulations  or  direct  feedback  from  professionals  in  the  field. 

•  Authenticity  of  role:  Learning  technologies  should  allow  instructors  to  stick 
to  their  role  as  facilitator,  freeing  them  from  the  need  to  invent  roles  to  guide 
and  evaluate  student  outcome.  This  can  be  seen  as  a  by-product  of  the  first 
two  areas;  new  technologies  allow  primary  sources  of  data  and  feedback  from 
professionals  in  the  field  to  fulfill  the  role  otherwise  invented  by  the 
instructor. 

In  the  next  section,  we  introduce  the  selected  lesson  content  domain  for  the  Phase  I  effort  and 
show  how  these  three  key  areas  of  authenticity  are  addressed. 

3.3  Lesson  Content  -  Interpreting  Aviation  Weather  Reports 

The  selected  lesson  content  for  this  eClass  prototype  was  the  Interpretation  of  Aviation 
Weather  Reports.  This  is  one  section  of  the  Military  Occupational  Speciality  (MOS)  93P10 
Aviation  Operations  Specialist  training  course.  This  content  was  selected  for  the  following  key 
features: 

•  it  has  a  well  defined  structure  that  can  me  modeled  in  the  eClass  format 

•  the  content  can  be  tested  using  multiple  choice  questions 
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•  subject  matter  experts  are  readily  available  for  course  evaluations 

•  computer  based  training  for  this  content  was  not  yet  developed  by  the  Army 

•  the  content  would  benefit  from  multimedia  such  as  graphics,  audio  and  video 
As  illustrated  below  in  figure  3.1-1,  a  typical  weather  report  can  be  divided  into  10 


sections: 


Lootio&IP  ViaA  YiltVaity 
MITABIXSG  24025SP45  «4SK  ICTZkK 


iaiaay  YtfvriPwi 
S32i/i2oorr  BicHoeo 


T<my  Altumtr 

OimOl  A2932 


SUKC 10006 VB 10 


Figure  3.1-1:  Aviation  Weather  Report  Sections 


With  this  natural  content  structure,  the  lesson  can  be  organized  into  discrete  content  areas 
that  the  student  can  navigate  through,  and  that  the  coaching  agent  can  monitor  and  provide 
guidance  on.  This  content  is  evaluated  according  to  the  three  key  authenticity  concepts  outlined 
in  the  previous  section: 

•  Authenticity  of  content:  The  Aviation  Weather  module  provides  an  excellent 
example  of  how  this  can  be  accomplished.  The  navigational  menu  used 
throughout  this  site  is  an  actual  weather  report  with  hyperlinks  placed  over 
each  key  data  field.  A  primary  source  of  data  is  used  in  the  design,  preparing 
students  for  what  they  will  see  in  the  field. 

•  Authenticity  of  outcome:  The  Aviation  Weather  training  is  based  on  actual 
MOS  93P10  Aviation  Operations  Specialist  training  content  that  is  used  in 
Army  National  Guard  training  around  the  country.  Exams  and  quizzed 
administered  by  the  eClass  agents  are  the  same  ones  given  to  students  at  Army 
training  locations. 

•  Authenticity  of  role:  The  agent  can  function  as  a  personal  assistant  for  the 
instructor.  One  example  of  this  is  its  ability  to  administer  and  grade  evaluative 
material.  In  addition,  the  agent  developed  for  this  content  is  able  to  answer 
student  questions  and  provide  feedback,  therefore  taking  the  load  off  the 
instructor.  Professors  using  the  Internet  for  distance  learning  need  to  be  freed 
from  this  role  so  they  can  focus  on  the  content  of  their  course.  Teachers  who 
cannot  fulfill  the  role  of  technical  guru  might  be  convinced  to  use  this  method 
of  educational  delivery  if  agent  technology  can  mitigate  the  need  to  seek 
outside  help. 
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4.  System  Architecture 

4.1  Electronic  Classroom  Architecture 

The  architecture  of  our  agent-enhanced  eClass  is  shown  below  in  figure  4,1-1.  Here,  we  have 
designed  and  implemented  a  Java-based  eClass  agent  to  run  on  the  student's  local  web  browser 
and  resides  within  the  eClass  HTML  interface.  As  shown,  this  client-side  eClass  agent 
communicates  back  to  an  NT  server  to  access  the  learning  capabilities  of  Learn  Sesame  in 
addition  to  storing  and  accessing  knowledge  of  the  students  and  the  lesson  content. 


Figure  4.1-1:  Agent-Enhanced  eClass  Architecture 


The  eClass  agent  is  the  heart  of  the  eClass  architecture  and  controls  all  that  the  student  and 
instructor  interact  with.  It  includes  the  real-time  inferencing  capabilities  and  keeps  a  local  copy 
of  its  high  level  training  and  administration  knowledge  within  an  Agent  Knowledge  Base  (KB). 
All  knowledge  and  data  that  the  agent  learns  is  stored  on  the  server  for  long-term  access.  To 
accomplish  the  Java  to  ODBC  communication,  we  take  advantage  of  Allaire's  Cold  Fusion 
software.  The  Learn  Sesame  intelligence  engine  is  provided  with  continual  user  events  by  the 
agent.  Since  the  Learn  Sesame  engine  stores  its  own  history  of  events,  it  can  provide  immediate 
feedback  to  the  agent  if  it  learns  a  new  fact  any  time  an  event  is  processed. 

4.2  Electronic  Classroom  Interface  Design 

The  Aviation  Weather  eClass  prototype  employs  a  three  part  frames  structure,  which 
includes  the  agent  coach  area,  the  content  whiteboard,  and  the  student  navigation  workdesk. 
Figure  4.2-1  illustrates  this  three  frame  HTML  eClass  interface: 
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Figure  4.2-1:  eClass  Interface 


This  has  technical  advantages,  which  will  be  discussed  below,  but  also  makes  for  a  more 
intuitive  user  interface.  Each  frame  window  has  a  clearly  defined  purpose  for  the  user; 

•  Coaching  Agent  frame:  This  is  the  upper  left  portion  of  the  browser  window 
and  is  reserved  for  the  Java  applet  that  “speaks”  to  the  database  and  contains 
the  rules  for  coaching  and  monitoring  student  progress  and  performance.  In 
the  weather  module,  this  is  the  place  where  quizzes  and  quiz  results  are 
offered  along  with  a  place  to  ask  questions,  an  opportunity  to  take  notes  and  a 
link  to  the  overview  of  the  course  content.  This  last  item  functions  as  a  panic 
switch  that  brings  students  back  to  the  top  of  the  lesson  content  in  case  they 
get  lost  or  disoriented. 

•  Content  Whiteboard  frame:  This  is  the  large  frame  window  to  the  right  in 
the  browser  window.  This  is  where  the  main  body  of  a  lesson’s  content  is 
displayed.  Care  was  taken  to  make  this  area  as  large  as  possible,  thereby 
reducing  the  need  to  scroll  and  improving  readability. 

•  Student  Navigation  frame:  This  is  the  short  horizontal  frame  window  that 
traverses  the  bottom  portion  of  the  browser  window.  This  is  where  a 
navigational  menu  is  displayed  to  the  user  that  brings  together  the  entire 
lesson.  The  design  of  this  navigational  structure  as  implemented  in  the 
Weather  and  WebCraft  modules  represents  an  effort  to  convey  where  in  the 
lesson  the  student  is  and  how  much  information  the  overall  site  contains.  A 
yellow  highlight  appears  on  the  menu  item  that  corresponds  to  the  information 
the  student  is  seeing  in  the  content  frame. 
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The  navigational  issues  addressed  in  this  design  are  perhaps  the  most  noteworthy.  The 
problem  of  place  and  context  has  plagued  the  web  development  community  for  some  time.  Sarah 
Horton  and  Patrick  Lynch,  Directors  of  the  Yale  Center  for  Advanced  Instructional  Media,  point 
out  the  challenge  of  providing  effective  navigation  through  a  web  site  is  in  imparting  a  sense  of 
location: 

Readers  need  a  sense  of  context,  of  their  place  within  an 
organization  of  information.  In  paper  documents  this  sense  of 
‘where  you  are’  is  a  mixture  of  graphic  and  editorial  cues  supplied 
by  the  graphic  design  of  the  book,  the  organization  of  the  text,  and 
the  physical  sensation  of  the  book  as  an  object. 

Horton,  Lynch,  Web  Style  Guide,  1997. 

http://info.med.yale.edu/caim/manual/interface/navigation.html 
4.2  eCIass  Design  Advantages 

Contextual  placement:  Course  content  is  supported  by  an  intuitive  navigational  scheme  that 
conveys  a  sense  of  where  in  the  lesson  a  particular  element  resides.  At  the  same  time,  the  frames 
interface  prevents  navigational  stmctures  from  getting  in  the  way  of  content. 

Ease  of  use :  Navigation  is  possible  no  matter  where  a  student  has  scrolled  to  in  the  content 
frame.  This  eliminates  a  step  and  reduces  the  potential  for  frustration. 

Flexible  sequencing:  Training  materials  should  have  linear,  rigid  paging  functions.  Limiting 
links  to  Next  and  Previous  pages  is  a  common  practice.  Extraneous  links  to  outside  resources 
only  confuses  the  overall  message  of  a  lesson.  However,  students  who  return  to  the  same  lesson 
for  retraining  or  to  reinforce  the  message  of  a  lesson  find  this  too  restrictive.  The  design  of 
navigational  elements  in  the  Weather  eClass  provides  a  clear  sequence  for  students  to  follow  in 
learning  the  skill,  but  also  allows  a  great  amount  of  flexibility.  Students  may  skip  from  one 
section  of  the  lesson  to  another  without  losing  place  or  context. 

Consolidation  of  functionality:  A  significant  advantage  to  instmctors  and  content  providers 
seen  in  the  prototype  eClass  is  an  integration  of  key  course  activities.  Questions  can  be  asked  by 
students  and  answered  by  the  agent  without  the  need  to  launch  any  outside  software  application. 
Email  messages  are  automatically  sent  to  the  content  provider  when  questions  cannot  be 
answered  using  the  content  in  the  agent  database.  The  frames  structure  allows  as  many  as  three 
separate  pages  of  information  to  be  displayed,  allowing  for  more  opportunities  to  relate  one  piece 
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of  information  to  another  or  to  an  alternate  medium  such  as  sound  or  video,  or  to  a  larger 
context  held  in  an  outside  resource. 

Multimedia:  Low  bandwidth  solutions  are  used  to  deliver  both  audio  and  video  sources  of 
information.  The  Weather  eClass  makes  use  of  sound  files  (.au)  to  simulate  how  weather 
personnel  would  speak  data  to  a  pilot. 
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5.  Prototype  Implementation 

5.1  Implementation  Design 

A  Java  applet  working  in  conjunction  with  a  database  can  be  a  very  powerful  tool  for 
educators  interested  in  web  based  instruction.  Agent  technology  as  used  in  this  Phase  I  project 
provides  a  viable  means  of  tracking  student  involvement  in  each  lesson,  scheduling  and 
administering  of  student  evaluations,  and  answering  student  questions. 

The  technology  used  to  bring  coiuse  content  together  with  a  database  and  user  preferences 
involves  creation  of  a  Java  applet.  The  agent  developed  in  this  contract  can  be  seen  as  consisting 
of  three  elements:  a  database,  a  Java  applet,  and  the  HTML  files  that  make  up  a  course  web  site. 
Using  JavaScript  within  the  HTML  files,  data  objects  are  created  that  can  be  passed  to  the  applet 
and  in  turn  stored  in  the  database.  In  the  quizzing  area,  for  example,  student  responses  are  passed 
to  a  JavaScript  that  invokes  a  function  in  the  applet  specifically  designed  to  accept  the  answers 
and  categorize  them  as  pass  or  fail.  This  same  process  can  be  used  in  many  ways.  Each  time  a 
student  visits  a  page  by  clicking  on  a  link,  a  JavaScript  function  is  activated  that  requests  the 
page  from  the  Java  coaching  agent,  who  then  logs  that  activity  in  the  database. 

5.2  Prototype  eClass  System  Requirements 

To  limit  the  testing  requirements  for  the  prototype,  the  target  client  platform  was  chosen  as 
the  following: 

•  Netscape  3.0  or  higher  running  on  Windows  95/NT 
The  prototype  server  requirements  are  the  following: 

•  Windows  NT 

•  Microsoft  Information  Server 

•  Allaire  Cold  Fusion  ODBC  connection  server 

The  prototype  eClass  client  can  also  be  viewed  on  Macintosh  operating  systems  running 
Netscape  3.0,  however  there  are  known  problems  with  the  Gold  version  of  Netscape  that  may 
cause  errors  while  running  eClass. 

5.3  Student  Logon 

The  developed  prototype  can  be  accessed  at  http://agents.cra.com/eclass/weather ,  and  will 
prompt  the  student  for  a  name  and  password  as  illustrated  in  figure  5.3-1: 
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Figure  5.3-1:  eClass  Agent  Interface 

If  the  agent  has  not  seen  the  particular  student  before,  it  will  ask  for  a  password.  When  the 
student  enters  his/her  own  personal  password,  the  agent  automatically  creates  a  new  account  for 
that  student. 


5.3  eClass  Course  Overview 

After  a  student  enters  the  agent-enhanced  eClass,  he/she  is  free  to  navigate  throughout  the 
content  at  their  own  pace.  The  agent  assists  the  beginner  by  presenting  the  content  overview  in 
the  content  whiteboard  frame  as  shown  in  figure  5.3-2: 
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distributed  via  tetetypevriter  or  computer.  Hovaver,  since  pitots 
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Figure  53-1:  Weather  eClass  Content  Overview 
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Students  can  then  step  through  the  content  overview  using  the  large  blue  navigation  arrow. 
When  the  student  has  completed  the  overview,  the  coaching  agent  guides  the  student  to  the  first 
section  of  the  weather  report.  This  serves  to  get  the  student  acquainted  with  the  eClass  system, 
while  introducing  the  weather  content  material. 

5.4  Agent  Administration  of  Exams 

The  coaching  agent  monitors  the  student’s  progress  through  the  course  and  administers  the 
exams  as  it  determines  when  the  student  is  ready  for  a  quiz  (figure  5.4-1). 


Figure  5.4-1:  Agent  Automatically  Administers  Quiz 

The  coaching  agent  then  scores  the  student’s  results,  providing  feedback,  and  guiding  the 
student  through  additional  instruction.  The  coaching  agent  insures  that  the  student  is  not 
overloaded  with  exams  and  allows  the  student  to  advance  only  after  successfully  completing 
lower  level  exams. 

The  completed  prototype  contains  training  material  completely  supported  by  the  eClass 
coaching  agent.  The  training  consists  of  the  following  quiz  material: 

•  Quiz  on  weather  codes 

•  Quiz  on  Wind  section 

•  Three  intermediate  level  Practical  Exercises 

•  Two  advanced  level  Practical  Exercises 

Figures  5.4-1  and  5.4-2  illustrate  a  sample  quiz  and  exam  that  is  administered  by  the 
coaching  agent: 
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Figure  5.4-1:  Weather  eClass  Codes  Quiz 
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Figure  5.4-2:  Weather  eClass  Sample  PE  Exam 


As  illustrated,  the  eClass  agent  administers  quizzes  and  exams  in  the  form  of  multiple  choice 
questions.  The  exams  are  brought  to  the  student  only  after  they  have  looked  at  the  corresponding 
content  material.  Students  are  allowed  to  return  to  course  content  before  completing  the  exam  to 
review  additional  material.  If  so,  the  agent  continues  to  bring  the  incompleted  exam  to  the 
student  until  they  successfully  complete  it. 

After  completing  a  quiz  or  exam,  the  coaching  agent  reviews  the  student's  answers  and 
analyzes  the  results.  Based  on  the  attributes  of  each  individual  question,  the  agent  then 
determines  which  content  area  the  student  requires  additional  guidance  through.  A  sample  quiz 
result  is  shown  in  figure  5.4-3: 
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Figure  5.4-3:  eClass  Agent  Exam  Review  and  Guidance 


By  clicking  on  the  blue  links  in  the  quiz  results,  the  student  is  automatically  guided  to  the 
content  material  that  they  had  difficulty  with. 

This  quiz  administration  capability  requires  that  preset  rules  be  defined  for  the  agent  to 
follow.  Using  a  simple  test  for  conditionality,  the  agent  accesses  these  rules  to  determine  when 
an  evaluation  should  be  given.  In  the  programming  of  the  applet,  each  answer  has  been 
associated  with  the  Uniform  Resource  Locator  (URL)  of  the  correct  answer.  A  wrong  answer 
invokes  this  association  and  the  student  is  given  a  hypertext  link  for  where  to  obtain  the  correct 
information. 


5.5  Content  References  and  Related  Links 

The  prototype  eClass  Agent  provides  the  student  with  on-demand  references  and  links  to 
assist  in  the  learning  process.  This  is  illustrated  in  figure  5.5-1: 
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Figure  5.5-1:  References  and  Related  Links 

This  demonstrates  how  the  eClass  agent  can  maintain  a  consistent  source  of  reference 
material  for  the  student  to  review  at  any  point  in  the  lesson  content. 

5.6  Agent  Question  and  Answer  Capabilities 

A  very  useful  technical  feature  of  this  prototype  is  the  ability  for  students  to  ask  questions  of 
the  agent  and  for  the  agent  to  answer  these  questions  using  a  knowledge  base  supplied  by  the 
instructor  (figure  5.6-1).  The  student  simply  types  in  a  question  using  standard  English  language, 
and  the  agent  identifies  the  corresponding  answer  based  on  filtered  keywords. 
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Figure  5.6-1:  Q&A  Agent  Interface 


The  agent  functions  as  a  facilitator,  storing  questions  it  cannot  answer.  These  are  saved  to  an 
administrator’s  page,  usually  for  the  instructor’s  eyes  only  (although  students  might  also  be 
allowed  to  view  this  if  needed).  Once  the  instructor  (or  any  knowledgeable  content  provider) 
supplies  the  answers,  the  agent  emails  them  to  the  students  who  asked.  It  then  stores  the  answers 
in  the  database  for  future  use. 


Figure  5.6-2:  Q&A  Agent  Email  Capability 

The  cumulative  effect  of  this  two-way  interaction  is  a  highly  adaptive  system  that  gets  better 
with  time.  As  the  instructor  answers  more  questions,  the  knowledge  base  grows,  and  the  agent  is 
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better  able  to  provide  immediate  answers  to  students.  Instructors  are  free  to  answer  questions 
asynchronously  while  students  reap  the  benefits  of  a  synchronous  answer  bank. 

In  addition,  instead  of  questions  going  to  a  single  individual  and,  as  a  result,  the  knowledge 
base  comprising  the  perspective  of  one  person,  an  entire  team  of  instructors  can  use  the  web 
based  interface  to  collaboratively  extend  the  database  with  the  collective  experiences  and 
expertise  of  a  group. 

5.7  Agent  Maintains  Student  Notes 

Figure  5.7-1  illustrates  the  student  course  notebook  which  is  available  to  all  students  using 
the  electronic  classroom.  The  notebook  allows  the  student  to  keep  a  collection  of  notes  relating 
to  the  course  content,  and  have  them  available  at  any  point  throughout  the  course  lesson. 


MITABrCSO  24025SQ45  1/4SII  WIZBA  5321/1200?!  BKB080  Ol/MOl  A2932  SMKCIG006V010 


Figure  5.7-1:  Student  Course  Notebook 

The  coaching  agent  stores  the  student  notes  inside  the  MS  Access  database  on  the  server,  and 
displays  the  notes  to  the  student  inside  the  content  frame  area  when  the  student  clicks  on  the 
Note  button  within  the  Agent  frame  area.  The  student  can  then  read  or  modify  their  notes  and 
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then  click  the  Save  button  to  have  the  coaching  agent  save  the  notes  for  the  student.  The  saved 
notes  are  then  available  to  the  student  each  time  they  enter  the  classroom. 

5.8  Agent  Multimedia  Capabilities 

The  prototyped  agent-enhanced  electronic  classroom  has  been  developed  with  graphic,  audio, 
and  video  multimedia  capabilities.  Figure  5.8-1  demonstrates  the  use  of  weather  graphics  for 
student  reference. 


Figure  5.8-1:  eClass  Graphics  Capability 


Audio  capabilities  have  been  fully  utilized  throughout  the  weather  training  class  as  illustrated 
in  figure  5.8-2.  When  a  student  moves  the  mouse  over  a  speaker  graphic,  it  hilights  and  runs  an 
associated  audio  file  when  clicked. 
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Figure  5.8-2:  eClass  Audio  Capability 


For  this  eClass  prototype,  the  audio  files  are  used  to  allow  a  student  to  hear  how  a  coded 
portion  of  the  weather  report  is  spoken  by  an  operator.  The  video  capabilities  were  also 
prototyped  within  eClass  and  made  available  for  investigating  external  distance  learning 
programs  as  illustrated  in  figure  5.8-3: 
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Figure  5.8-3:  eClass  Video  Capability 


The  eClass  prototype  requires  the  VivoActive  plug-in  to  view  video  files  on  Windows 
operating  systems.  If  the  student's  browser  does  not  have  the  VivoActive  plug-in  installed,  then 
the  agent  will  guide  the  student  to  where  they  can  download  a  version  over  the  internet. 
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6.  Prototype  Evaluation 

The  developed  Agent-Enhanced  Electronic  Classroom  (eClass)  was  evaluated  for  its 
usefulness  within  the  Weather  training  course  and  CyberEd  WebCraft  course.  Both  evaluations 
were  found  to  be  very  successful,  with  a  number  of  system  improvement  recommendations. 

6.1  Aviation  Weather  Training  Evaluation  by  Army  National  Guard 

The  prototype  eClass  for  interpreting  aviation  weather  reports  was  evaluated  as  part  of  the 
Army  National  Guard  aviation  training  course  in  Ft.  Indiantown  Gap  in  Pennsylvania.  Two  sets 
of  students  used  the  eClass  training,  each  having  access  to  the  web-based  course  for  two  days  out 
of  their  three  week  training  period. 

The  first  group  of  students  were  used  for  initial  prototype  testing,  and  served  well  to  identify 
the  following  areas  of  improvement: 

•  The  coaching  agent  needed  to  bring  quizzes  to  students  much  faster  since  they  were 
getting  preoccupied  during  self-paced  lesson  navigation. 

•  Students  needed  the  ability  to  tell  the  coaching  agent  to  administer  an  exam  whenever 
the  student  requested  one. 

•  The  initial  dbANYWHERE  database  connection  was  not  able  to  keep  up  with  the  fast 
pace  at  which  students  would  click  on  the  course  material.  Therefore,  a  new  Allaire 
Cold  Fusion  database  connection  was  required. 

The  second  group  of  Army  National  Guard  students  were  given  access  to  the  completed 
prototype  that  was  enhanced  according  to  the  results  of  the  first  group’s  evaluation.  Since  there 
were  only  six  students  in  this  group,  statistical  averages  would  not  be  significant.  However, 
simple  plots  of  the  student  responses  do  indicate  the  overall  success  of  the  eClass  Weather 
training.  Appendix  A  presents  the  responses  of  the  six  students  to  a  survey  completed  by  them 
after  taking  part  in  the  Weather  eClass  training  using  the  developed  prototype. 
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Figure  6.1-1;  How  useful  did  you  find  the  Internet  training? 

A  sample  of  the  student  evaluation  results  is  illustrated  in  figure  6.1-1,  showing  that  all  the 
students  found  the  training  to  be  useful  and  effective  within  the  agent-enhanced  eClass  internet 
training.  The  evaluation  results  can  be  summarized  in  the  following  areas: 

•  Students  were  somewhat  familiar  with  the  aviation  weather  report  before  using  the 
eClass  prototype. 

•  Most  of  the  students  only  had  one  training  session  with  the  eClass  prototype. 

•  Students  spent  1  to  2  hours  using  the  prototype. 

•  The  majority  of  smdents  found  the  eClass  weather  training  to  be  very  useful. 

•  All  students  had  little  to  no  difficulty  using  the  eClass  weather  training  course. 

•  Students  felt  they  learned  a  bit  more  about  aviation  weather  after  using  the  prototype. 

•  There  was  a  wide  range  of  internet,  computer,  computer-based  training,  and  email 
experience  between  the  Army  National  Guard  students. 

•  The  students  had  difficulty  with  the  eClass  sound  capabilities,  due  mainly  with  local 
hardware  problems. 

•  All  students  felt  the  material  was  organized  very  well  and  course  navigation  was 
excellent. 

•  Students  were  satisfied  with  the  quality  of  the  quizzes,  though  it  was  clear  that  some 
improvement  was  desired. 

•  There  was  plenty  of  content  information  for  students  to  leam  from. 

•  The  Q«&A  Agent  provided  accurate  answers  to  all  student  questions. 

•  Most  students  thought  the  agent’s  guidance  was  helpful,  though  there  is  room  for 
improvement. 
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•  Most  students  felt  the  agent  could  have  done  better  in  bringing  quizzes  to  them  when 
they  were  ready. 

•  Most  agreed  that  it  was  helpful  to  have  the  agent  always  available  to  them. 

•  Students  found  it  easy  to  get  started  using  the  eClass  prototype  and  that  the  interface 
was  easy  to  work  with. 

This  summarizes  the  training  evaluation,  with  the  complete  results  presented  in  Appendix  A. 
Additional  evaluations  were  performed  at  UMass  Dartmouth  using  students  taking  a  web-based 
course  through  their  CyberEd  program,  and  is  presented  in  the  next  section  along  with  an 
instructor’s  evaluation  of  the  prototype  eClass. 

6.2  eClass  Agent  Evaluation  by  UMass  Dartmouth 

The  eClass  frame  structure  and  Q&A  Agent  capabilities  were  evaluated  on  students  taking 
the  UMass  Dartmouth's  WebCraft  course.  The  eClass  agent  was  added  as  an  optional  link  from 
UMass  Dartmouth's  CyberEd  course  environment. 

Student  reactions:  Students  from  a  CyberEd  course,  reacting  informally  to  the  WebCraft 
eClass  agent  prototype,  all  pointed  out  that  the  area  to  ask  questions  of  the  agent  was  too  small. 
Despite  the  technical  limitation  of  only  being  able  to  match  a  few  words  to  the  content  of  the 
database  records,  future  development  efforts  might  place  the  question  form  in  the  content  frame 
and  give  more  explicit  instruction/explanation  of  how  the  Q&A  agent  works.  A  larger  area  to 
type  in  questions  was  requested  from  six  of  the  nine  students  who  responded  with  open 
feedback. 

Instructor  comments:  The  following  paragraphs  present  the  instructor’s  comments  and 
suggestions  regarding  the  eClass  structure  and  its  application  to  university  courses. 

Authoring  overhead:  The  instructional  design  using  this  process  takes  time  -  approximately 
five  to  six  hours  production  time  to  manufacture  a  single  hour  of  instruction.  This  represents  a 
dramatic  increase  from  the  three  hours  of  production  time  needed  to  yield  one  hour  of  material 
using  straight  HTML.  Future  development  efforts  might  be  to  develop  an  authoring  tool  kit  that 
makes  file  management  easier.  The  navigational  menus  are  easily  the  most  time  consuming 
element  of  this  process. 

More  real  world  practice:  Simulations  might  be  designed  where  students  are  asked  to 
evaluate  their  own  real  world  response  performance.  For  example,  in  the  Weather  module,  a 
scenario  could  be  devised  in  which  a  student  must  decode  a  weather  report  in  real  time  and 
provide  a  plain  English  script  for  radio  broadcast. 
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A  more  graphical  approach:  Many  of  the  textual  elements  used  for  navigation  can  be 
replaced  with  image  icons  or  icons  that  used  stylized  text  an  colors. 

More  multimedia:  As  noted  in  previous  sections,  sound  files  are  used  where  appropriate. 
However,  there  are  instances  where  video  or  audio  would  better  convey  a  message  than  the  text 
based  counterpart.  Development  of  multimedia  approaches  would  further  support  content  and 
attract  student  interest. 

Screen  real  estate:  The  frames  interface  might  be  extremely  difficult  for  users  who  have 
small  monitors.  A  non-frames  version  might  be  developed  that  places  the  agent  and  navigational 
elements  in  the  background.  Although  this  may  make  for  an  impoverished  environment  and  may 
negate  some  of  the  benefits  described  above,  an  alternate  design  might  offer  more  choices  to 
organizations  and  institutions  limited  in  hardware  resources. 

Storing  of  quiz  answers:  Currently,  students  must  re-take  an  entire  quiz  if  they  answer  any 
one  question  wrong  on  it.  A  better  technical  design  for  how  the  JavaScript  passes  answers  to  the 
Java  applet  is  needed  so  that  students  would  only  have  to  retake  the  portion  of  the  quiz  on  which 
they  performed  poorly. 

Reports  to  instructor:  Currently,  the  prototype  eClass  does  not  report  back  to  the  instructor 
about  student  progress.  While  students  may  look  at  their  quiz  results  in  the  Aviation  Weather 
module,  currently  the  eClass  coaching  agent  doesn’t  allow  the  instructor  to  view  that 
information. 
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7.  Phase  n  Requirements  And  Commercialization 

7.1  Incorporate  Learn  Sesame  Server 

The  Phase  I  prototype  used  Java-based  local  client  processing  to  perform  the  agent  analysis 
of  student  navigation  and  quiz  analysis.  This  enabled  a  fast  development  of  a  feasibility 
prototype  without  taking  advantage  of  our  full  learning  capabilities.  During  a  Phase  II  effort,  the 
Learn  Sesame  Server  will  be  required  to  perform  the  major  agent  processing  to  provide  the 
eClass  with  superior  student  customization. 

Use  of  the  Learn  Sesame  Server  will  require  architecture  modifications  to  take  full  advantage 
of  the  engine’s  learning  capabilities.  The  Java  eClass  agent  clients  will  communicate  with  a 
Learn  Sesame  Server  by  sending  CORBA  messages  containing  the  events  and  objects  to  be 
learned.  After  processing  the  events,  the  Learn  Sesame  Server  will  send  facts  back  to  the  eClass 
agents  through  the  CORBA  message  format. 

7.2  CD-ROM  Video  Storage  Requirements 

Computer  based  training  (CBT)  is  a  relatively  new  topic  that  large  companies  are  beginning 
to  explore.  According  to  Training  magazine’s  1996  Industry  Report,  roughly  40  percent  of  U.S. 
organizations  with  more  than  100  employees  already  use  CBT  for  some  training  applications. 
With  the  boom  in  multimedia  and  the  Internet,  corporations  are  investigating  how  they  can  use 
this  new  form  of  technology.  Using  the  Internet,  several  start-up  companies  are  providing  audio 
and  video  players  available  for  download.  As  Charles  River  Analytics  has  seen  in  the  on-line 
classroom,  the  quality  and  wait  time  for  video  can  become  tedious  for  our  users. 

Bypassing  Bandwidth  -  By  having  multimedia  clips  (and  other  unchanging  media)  on  CD- 
ROM,  users  no  longer  have  to  deal  with  bandwidth  limitations.  We  have  developed  eClass  for 
any  Internet  browser,  and  this  platform  independence  is  extremely  important.  Since  the  majority 
of  course  content  is  static,  multimedia  components  can  be  placed  on  a  supplementary  CD.  Even 
if  an  individual  does  not  have  an  actual  CD  player,  the  computer  will  be  attached  to  the  Local 
Area  Network.  An  administrator  can  then  place  the  CD  on  a  drive  accessible  to  the  rest  of  the 
students,  and  they  will  be  able  to  receive  much  faster,  higher  resolution  video  than  previously 
seen. 

Production  Costs  -  The  cost  of  producing  a  CD  is  fairly  inexpensive,  especially  if  blank  CD’s 
are  purchased  in  large  quantities.  This  CD-ROM  content  can  also  be  developed  as  a 
supplementary  aid  to  teachers  instead  of  a  stand-alone  application,  but  ultimately  our  market 
research  will  help  decide  what  content  needs  to  be  burned  onto  the  compact  disks. 
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The  future  of  this  media  is  uncertain;  within  the  last  year  there  has  been  much  talk  about 
DVD-ROM,  which  can  initially  hold  three  times  the  information  a  CD  does.  Just  as  the  CD’s  are 
replacing  audio  tapes,  it  is  important  that  we  investigate  providing  new  content  via  this  new 
format. 


7.3  Requirements  for  Chat  Rooms 

Although  the  prototype  eClass  has  many  interactive  feamres,  a  chat  room  is  a  requirement  for 
distance  learning.  According  to  Training,  “What  makes  classroom  training  hard  is  that  you  have 
to  adapt  on  the  fly,  in  real  time,  to  different  learning  styles.  A  good  live  instructor  can  do  that. 
A  multimedia  program  can’  t”[  1  ] . 

Interactivity  -  The  teacher  and  student  can  converse  in  real  time.  Therefore,  students  no 
longer  just  read  content  on  the  screen,  but  also  have  an  opportunity  to  ask  questions  of  their 
professor.  Other  students  may  also  join  a  chat  forum  on  a  particular  topic,  making  this 
component  multi-purposed.  Charles  River  Analytics  will  look  to  purchase  one  already 
developed  by  another  software  company.  Potential  customers  can  then  be  shown  how  this  chat 
room,  along  with  electronic  mail,  can  prove  to  be  a  very  useful  addition  with  distance  learning. 

Lotus  Example  -  A  current  company  using  an  on-line  course  module  is  Lotus  Development 
Corporation.  This  software  giant  supplements  conventional  teaching  methods  with  Internet 
training.  Explaining  the  variations  in  learning,  Nancy  Leo,  manager  of  curriculum  development 
says: 

“Because  technology  is  technology  and  people  are  people,  learners  are 
differentiated  in  other  ways.  There  are  people  who  say,  T  have  to  go  to  a 
class.’  There  are  people  who  say,  ‘Don’t  bother  me.  I’ll  learn  it  when  I 
need  it.’  “There  are  people  who  say ,  ‘Just  give  me  the  documents’’. 

Leo  says  that  multimedia  is  most  effective  for  “discovery  learners’’,  but  that  “those  people 
are  not  in  the  majority”[l].  Therefore,  most  of  training  at  Lotus  continues  to  be  held  in  a 
classroom  and  instructor  based.  Successful  multimedia  based  distance  learning  is  trying  to  find 
the  right  combination  of  software  to  meet  the  most  general  audience,  and  the  ability  to  talk  with 
one  another  is  a  substantial  component. 

7.4  Requirements  for  a  Progress  Indicator 

Currently,  students  in  our  on-line  classroom  may  select  “Quiz  Results”  to  see  which  quizzes 
they  have  passed.  Hyperlinks  from  each  quiz  brings  the  student  back  to  a  specific  chapter  he  or 
she  need  to  focus  on  if  they  had  not  passed  the  quiz.  We  need  to  build  upon  and  add  new 
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features  that  are  still  accessible  to  any  student  using  a  current  Internet  browser  (Netscape 
Navigator  or  Internet  Explorer).  This  new  object,  renamed  "Progress  Indicator,"  shows  the 
student  all  of  the  relevant  information  and  how  he  or  she  is  currently  doing  on  a  particular 
course.  To  demonstrate  these  new  components,  we  investigated  the  on-line  SEC  Series  7  exam. 
A  student  clicks  on  Progress  Indicator  from  the  main  menu.  Once  inside,  he  or  she  sees  the 
following:  Student  Information,  SEC  Content  Covered,  Agent  Advice,  Quiz  Results,  and 
Instructor  Remarks. 

Student  Information  -  As  with  the  other  newly  introduced  components.  Student  Information 
will  be  interacting  with  an  SQL  database,  namely  Informix  for  Windows  NT.  Data  such  as  the 
course  length,  student  e-mail  address  and  other  relevant  facts  will  reside  here.  The  student  is 
allowed  to  change  only  his  or  her  e-mail  address  from  this  region. 

SEC  Content  Covered  -  This  component  provides  the  user  with  a  graphical  representation  of 
the  sections  they  have  covered.  This  Java-based  applet  will  be  developed  to  interact  with  our 
Cold  Fusion  database  server.  The  student  can  see  different  color  coded  sections  showing  how 
they  are  doing  individually  on  the  course  material.  Since  we  are  already  tracking  every  page  the 
user  visits,  this  new  applet  will  be  easy  to  implement.  The  student  has  control  over  the  colors  by 
using  a  mouse  to  double  click  on  the  applet. 

Quiz  Results  -  This  component  is  similar  to  the  SEC  Content  Covered  in  that  students  will 
see  how  many  quizzes  remain  and  what  has  been  completed.  The  colors  and  graphics  used  are 
visually  appealing  to  the  student,  so  information  does  not  appear  too  static.  Audio  clips  can  even 
be  attached  to  each  quiz  successfully  passed  in  order  to  entice  the  student  to  continue  his  or  her 
progress.  From  the  Quiz  Results  section,  students  can  click  on  any  of  the  quizzes  listed  to 
immediately  be  tested. 

Agent  Advice  -  Agent  Advice  automatically  recommends  the  reviewing  of  certain  areas  if  the 
student  has  been  having  trouble  on  the  quizzes.  Since  this  feature  has  already  been  developed,  it 
will  be  incorporated  into  the  Progress  Indicator  with  the  other  components.  Students  can  click  on 
a  hyperlink  to  review  the  topics  suggested,  and  our  on-line  classroom  continues  to  prompt  them 
when  it  is  time  for  another  quiz. 

Instructor  Remarks  -  This  text  field  is  where  the  instructor  can  review  each  student’s 
progress  and  write  individual  or  class  notes  available  to  him  or  her.  A  picture  of  the  professor  as 
well  as  the  opportunity  to  respond  to  his  or  her  comments  via  e-mail  will  also  be  available 
through  the  Progress  Indicator  page. 
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Please  refer  to  Figure  1  for  the  complete  Progress  Indicator  layout  as  viewed  by  a  Netscape 
Navigator  user.  Also  note  that  any  number  of  the  above  components  can  be  removed  or  placed 
elsewhere  within  the  Progress  Indicator.  The  diagram  represents  the  full  featured  applets  in  a 
Hypertext  Markup  Language  document  using  a  Java  enabled  web  browser.  JavaScript  continues 
to  check  if  the  system  meets  the  given  requirements  before  continuing  into  the  on-line  classroom. 

7.5  Desktop  Video  Conferencing 

In  addition  to  the  chat  application,  it  is  necessary  that  Charles  River  Analytics  investigate  the 
possibilities  of  desktop  video  conferencing.  Since  students  may  be  located  all  over  the  world, 
face-to-face  contact  is  extremely  difficult  without  video  conferencing.  Although  the  technology 
is  in  its  infancy,  video  conferencing  software  is  yet  another  way  for  students  to  ask  teachers  and 
other  classmates  questions  about  the  course  they  are  studying. 

There  are  several  companies  offering  free  conferencing  software  for  the  Internet,  including 
Microsoft  Corporation  and  White  Pine  Software.  Microsoft’s  Net  Meeting  is  ideal  for  distance 
learning  in  that  it  offers  decent  frame  rates,  whiteboard  collaboration,  and  file  transferring.  The 
file  transferring  allows  teachers  and  students  to  download  documents,  audio  files,  and  any  other  i 

relevant  course  information  onto  his  or  her  individual  computer.  Whiteboard  sharing  lets  two 
users  exchange  ideas  over  a  blank  white  screen,  while  still  keeping  the  video  conferencing  intact. 

Although  two-way  digital  video-conferencing  has  bandwidth  limitations,  it  still  can  be  used 
as  another  way  to  convey  a  message  to  students.  “As  advanced  fiber-optic  networks  and 
improved  compression  techniques  become  widespread,  this  technology  is  likely  to  play  a  more 
significant  training  role”  [2].  Because  video-conferencing  is  still  in  its  infancy,  many  companies 
have  not  seriously  weighed  the  benefits.  This  technology  is  primarily  used  as  another  means  to 
convey  information  to  the  student.  Both  instructors  and  students  will  have  to  spend  more  time 
learning  how  to  use  the  software,  but  when  multimedia  interaction  between  teacher  and  student 
is  needed,  Charles  River  Analytics  will  not  overlook  this  useful  component  to  our  on-line 
classroom. 

An  example  use  of  video  conferencing  software  would  be  a  student  who  has  trouble 
understanding  a  chapter  from  our  on-line  classroom.  He  then  meets  with  his  instructor  at  a 
designated  time,  while  they  converse  using  a  digital  camera  and  speaker.  Using  a  combination 
of  audio,  video,  and  graphics  the  student  then  has  a  better  understanding  of  his  difficulties. 

By  using  software  already  in  existence,  Charles  River  Analytics  does  not  need  develop  this 
technology  in-house.  Once  acquired,  our  video  conferencing  component  is  yet  another  learning 
tool  that  can  be  used  to  market  to  our  potential  customers.  The  only  equipment  that  our 
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customers  may  not  have  are  the  digital  cameras,  as  they  are  not  standard  with  most  new 
computers  being  shipped.  Steve  Willett,  founder  of  Prosoft  distance  learning  software  says: 

“An  instructor  brings  a  whole  persona  that  can  never  be  emulated  by  a  machine. 
Everyone  who  comes  through  a  training  classroom  thinks  in  a  different  way  and 
wants  to  go  down  a  different  avenue  with  the  content”  [1]. 

By  introducing  video-conferencing,  this  is  another  option  corporations  can  use  to  supplement 
the  learning  process.  As  with  our  other  components,  we  must  deal  with  each  client  on  a  case  by 
case  basis  and  look  at  their  needs  to  ensure  a  successful  distance  learning  software  solution. 

7.6  Developing  a  Business  Model 

There  are  several  different  business  models  currently  being  used  which  are  profitable  on  the 
Internet.  In  a  Marketing  Management  article  by  Bruce  H.  Clark,  he  details  the  reasons  to  use 
each  plan  for  generating  revenue.  However,  the  initial  business  plan  for  the  on-line  electronic 
classroom  consists  of  the  following  mix:  The  Cable  TV  Model,  The  Customization  Model,  and 
The  Subscriber  Model. 

The  Cable  TV  Model  -  Clark  says  that  a  company  would  “provide  a  selection  of  advertiser- 
supported  free  content  while  charging  customers  for  the  premium  content”  He  later  notes  that  “a 
further  advantage  of  this  model  is  that  the  free  content  acts  as  a  sampling  device  for  the  premium 
content.  Occasional  free  trials  of  the  premium  content  may  convert  visitors  into  subscribers”  [3]. 
Internet  users  who  “surf’  onto  Charles  River  Analytics  web  site  will  find  our  distance  learning 
section,  and  be  able  to  partake  in  a  working  demo.  After  filling  out  some  information  for  us,  we 
can  record  the  amount  of  users  looking  at  our  demo,  and  contact  them  a  short  time  later.  In  the 
Cable  TV  Model,  our  company  should  not  show  advertisements,  only  limited  free  content. 

The  Customization  Model  -  This  particular  business  model  will  “charge  customers  for 
content  that  is  customized  to  meet  their  preferences.  This  model,  also  relatively  new,  takes 
advantage  of  database  software  and  the  Web’s  low  marginal  cost  of  information  transmission” 
[3].  This  strategy  works  well  with  the  marketing  of  corporate  clients.  If  Charles  River  Analytics 
can  purchase  the  rights  to  use  material  for  a  classroom,  we  can  then  directly  market  to  Intranets. 
Each  client’s  needs  will  vary,  so  we  can  select  certain  components  to  fit  their  corporate  needs. 
The  database  software  we  use  to  track  students  and  visitors  to  the  web  site  already  exists, 
therefore  customizing  software  is  one  model  of  attracting  conmiercial  clients. 

The  Subscriber  Model  -  The  focus  of  this  model  is  to  charge  customers  for  the  content  they 
view.  Similar  to  the  Cable  TV  Model,  this  is  an  excellent  way  for  our  company  to  receive  profits 
from  single  users.  If  an  individual  is  preparing  for  the  SEC  Exam,  and  not  affiliated  with  any 
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company,  he  or  she  could  take  the  course  off  of  our  web  site  during  the  specified  times.  These 
customers  will  pay  us  for  a  flat  rate  to  use  the  classroom,  and  in  return,  have  access  to  all  of  the 
components  we  have  developed. 

Consequently,  the  above  business  models  blend  together.  We  can  effectively  market  to  the 
individual  Internet  user  and  corporate  Intranets,  as  well  as  offer  methods  for  negotiating  with 
content  providers. 

7.7  Commercial  Web  Delivery  Mechanisms 

As  most  serious  Internet  users  know,  bandwidth  throughput  will  play  an  important  role  in  the 
future  of  the  web.  Companies  who  want  to  deliver  information  to  their  employees  have  to  deal 
with  these  limitations.  “Although  interactivity  is  a  valuable  part  of  Intranet  technology,  many 
companies  have  yet  to  use  it.  In  fact,  most  Intranets  still  function  as  virtual  warehouses  of  static 
information”  [4].  Fortunately,  there  has  been  much  recent  development  in  this  new  area  of 
bandwidth.  The  increase  in  bandwidth  can  be  applied  to  both  our  stand-alone  users  as  well  as  the 
corporate  Intranet  market.  This  topic  is  investigated  because  Charles  River  Analytics  success  in 
distance  learning  depends  on  the  quantity  of  content,  and  the  speed  in  which  it  can  be  delivered 
to  our  customers.  The  following  modem  technologies  are  investigated:  ISDN,  Cables  Modems, 
and  ADSL. 

ISDN  -  ISDN  (Integrated  Standards  Digital  Network),  unlike  Cable  Modems  and  ADSL, 
have  been  around  since  the  1970’ s.  ISDN  is  currently  offered  by  several  of  the  large  telephone 
companies  for  a  premium  price,  but  offers  a  capacity  of  only  128  kpbs.  This  rate  is  terribly  slow 
when  compared  to  9  and  10  Mbps  offered  by  the  competitors.  The  only  major  advantage  ISDN 
users  have  is  that  it  is  a  solid  product  that  delivers  home  users  faster  access  rates  than  28.8  kpbs. 
Unfortunately,  the  high  maintenance  costs  and  slow  bandwidth  rates  can  force  this  technology  to 
become  obsolete  when  the  competition  gets  approval  from  the  FCC  and  telephone  companies. 

Cable  Modems  -  Cable  companies  are  investing  millions  of  dollars  to  develop  Internet 
access  through  box  top  sets.  These  modems  allows  for  users  to  receive  information  at  rates  as 
high  as  10  Mbps.  This  is  obviously  far  greater  than  today’s  modem  speeds  of  up  to  56  kpbs. 
There  are  two  general  applications  that  can  be  used  for  these  higher  speed  modem:  “interactive 
video  and  high  speed  data  communications”  [5].  The  downside  to  this  new  technology  being 
developed  is  that  users  must  share  this  capacity  with  others  who  are  located  on  the  network. 
Thus,  the  download  transfer  rate  can  become  significantly  less,  as  other  users  log  on. 

ADSL  -  ADSL  (Asymmetric  Digital  Subscriber  Line)  has  much  more  potential,  and  future 
use,  than  Cable  Modems.  “The  strength  of  ADSL  compared  to  other  high  speed  transmission 
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alternatives  lies  in  the  number  of  existing  telephone  lines  -  now  approaching  700  million  - 
compared  to  new  cabling  which  has  reached  comparatively  few  homes  and  no  small  businesses” 
[5].  The  customer  using  ADSL  technology  has  a  downstream  of  9  Mbps,  but  unlike  the  Cable 
Modem,  does  not  have  to  share  this  bandwidth  rate  with  others.  The  majority  of  U.S.  households 
and  businesses  have  phone  lines  in  place,  so  once  ADSL  is  implemented  by  the  telephone 
companies,  end  users  will  see  a  huge  difference  in  the  speed  of  accessing  data. 

Charles  River  Analytics  can  be  confident  that  our  distance  learning  software  will  become 
faster  in  time,  due  to  the  new  developments  in  modems  and  access  times.  ADSL  allows  for  our 
video  player  to  no  longer  look  “choppy”  or  compressed,  but  to  smoothly  flow  from  frame  to 
frame.  Therefore,  much  more  video  content  can  be  stored  at  the  corporate  web  sites.  Soon, 
video  will  not  need  to  placed  onto  a  CD-ROM,  but  can  be  stored  on  a  server  for  immediate 
retrieval.  This  reduces  our  costs  in  production,  and  allows  for  repeat  customers  to  ask  for  more 
video  content  in  their  particular  classroom.  The  faster  speed  of  the  Internet  will  enhance  our 
classrooms  in  all  aspects,  ranging  from  corporate  Intranets  to  stand-alone  customers  who  wish  to 
self-study  one  of  our  on-line  courses. 

7.8  Web  Television  Market 

Until  recently,  personal  computers  and  television  sets  were  thought  of  as  two  separate 
applications.  However,  within  the  last  few  years,  the  general  public  has  quickly  started  to  adopt 
the  Internet  as  a  new  medium  in  receiving  information.  Companies  such  as  WebTV  realized  that 
many  households  did  not  want  to  spend  thousands  of  dollars  buying  an  Internet-ready  computer. 
Engineers  realized  that  the  television  sets  can  be  easily  converted  into  an  Internet  browser  similar 
to  what  the  video  game  industry  had  accomplished. 

Predicting  the  next  few  years.  Marketing  Management  writes: 

“the  future  will  be  powerfully  affected  by  how  quickly  Internet  access  grows  for 
businesses  and  consumers  and  by  how  quickly  Internet  capacity  expands.  Audio 
has  already  arrived  on  the  Web,  and  full-motion  video  will  eventually  join  it, 
revolutionizing  what  we  can  do  on  Web  pages.  Simpler  Internet-access  devices 
such  as  WebTV  and  faster  Internet-access  technologies  such  as  cable-modems 
promise  a  broader,  faster,  richer  World  Wide  Web  in  the  future”  [3]. 

The  future  of  the  Internet  depends  on  how  many  people  can  access  the  networks  and  at  what 
speed  the  information  is  received. 

With  the  example  of  WebTV,  there  is  no  software  needed.  After  purchasing  a  WebTV 
Internet  terminal,  a  customer  uses  their  telephone  line  to  connect  to  WebTV’s  local  Internet 


43 


R96291 


Charles  River  Analytics 


Service  Provider.  The  near  future  will  bring  many  more  smaller,  easy-to-use  applications  for 
accessing  the  Internet.  Fortunately,  Charles  River  Analytics’  electronic  classroom  will  be 
compatible  with  these  new  technologies  such  as  WebTV,  because  of  the  programming  language 
used,  and  the  current  standards  the  software  industry  established. 
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8.  Summary,  Conclusions,  and  Recommendations 

In  this  chapter  we  summarize  our  findings  (section  5.1),  present  our  conclusions  (section 
5.2),  and  offer  recommendations  for  a  follow-on  Phase  11  effort  (section  5.3).  Finally,  in  section 
5.4  we  explore  the  commercialization  potential  of  the  core  technology  developed  in  this  effort. 

8.1  Summary  of  Phase  I  Effort 

Our  Phase  I  study  centered  on  the  development  and  feasibility  assessment  of  an  agent- 
enhanced  electronic  classroom  for  the  web.  Six  development  tasks  comprised  our  effort: 

•  Identification  of  State-Of-The-Art  in  Electronic  Classrooms 

•  Determination  of  Prototype  Lesson  Domain 

•  System  Architecture  Design 

•  Implementation  of  a  Limited-Scope  Prototype 

•  Demonstration  and  Evaluation  of  Concept  Prototype 

•  Requirements  Specification  and  Commercialization 

We  first  defined  the  Phase  I  problem  scope.  This  involved  consultation  with  the  sponsor, 
through  which  we  selected  the  interpretation  of  aviation  weather  reports  as  the  lesson  content 
for  the  prototype  electronic  classroom.  The  following  requirements  for  the  Phase  I  effort  were 
identified: 

•  The  lesson  content  must  have  inherent  structure  that  can  be  navigated  by  students. 

•  The  lesson  must  be  testable  through  multiple  choice  questions 

•  The  tutor  agent  must  monitor  student  progress  through  content 

•  The  tutor  agent  must  administer  exams  and  provide  coaching 

•  The  electronic  classroom  must  demonstrate  multimedia  capabilities 

•  The  electronic  classroom  must  be  extensible  to  other  lesson  content 

•  The  agent  tutor  must  facilitate  student-instructor  interaction  and  communication 

With  the  problem  scope  and  requirements  identified,  the  target  lesson  content  was  selected  to 
be  a  limited  subset  of  the  Army  National  Guard  Aviation  Training  course.  The  selected  subset  of 
aviation  training  was  Interpreting  Aviation  Weather  Reports.  This  content  provided  a  complete 
lesson  that  could  be  presented  in  its  entirety,  while  remaining  small  enough  to  complete  in  the 
Phase  I  timeframe. 
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The  agent-enhanced  electronic  classroom  (eClass)  architecture  was  chosen  such  that  it  would 
be  easily  extensible  for  additional  in-depth  content  and  to  other  lesson  domains.  To  accomplish 
this,  a  Java-based  client-side  implementation  was  used  along  with  Cold  Fusion  server-side 
connections  to  a  MS  Access  database.  All  relevant  lesson  content  information  is  stored  within 
MS  Access  tables  and  HTML  files,  while  the  Java  code  contains  the  agent  tutor  rules  and  general 
eClass  functionality.  In  this  manner,  lesson  content  can  be  easily  modified  and  replaced  without 
changing  the  basic  functionality  of  the  eClass  and  its  agent  tutor. 

The  eClass  architecture  was  prototyped  for  the  aviation  weather  report  training,  and  is 
available  at  http://agents.cra.com/eclass  for  demonstration  purposes.  The  results  obtained  from 
this  Phase  I  prototype  effort  can  be  summarized  as  follows: 

•  a  three-frame  approach  to  the  eClass  user  interface  provides  a  consistent  learning 
environment  in  which  there  are  designated  areas  for  student  self-paced  navigation, 
agent  tutoring  interaction,  and  lesson  content  presentation. 

•  the  eClass  client-server  architecture  is  capable  of  accommodating  a  large  range  of 
training  content,  providing  easy  customization  for  developers  and  simplified 
maintenance  for  instructors. 

•  the  Agent  tutor  is  effective  in  monitoring  student  navigation  through  course  content, 
capable  of  administering  exams  when  needed,  successfully  corrects  exams,  and 
provides  accurate  coaching  for  areas  the  student  had  difficulty  with. 

•  the  Question  &  Answer  Agent  component  of  the  prototyped  eClass  successfully 
provides  direct  and  accurate  answers  to  student  questions,  decreasing  the  load  on 
actual  course  instructors. 

•  the  Question  &  Answer  Agent  component  is  capable  of  serving  as  a  stand-alone 
service,  and  was  demonstrated  for  the  commercial  OpenSesame  web  site,  Charles 
River  Analytics  intranet,  and  UMass  Dartmouth  CyberEd  course. 

•  the  prototyped  eClass  for  Weather  training  was  evaluated  by  two  groups  of  Army 
National  Guard  students.  Feedback  from  the  first  group  of  students  was  used  to 
modify  the  agent  functionality  for  the  second  group.  The  second  group  of  students 
successfully  completed  the  eClass  training  using  the  on-line  prototype. 

Based  on  the  results  of  our  Phase  I  effort,  the  agent-enhance  eClass  architecture  has 
significant  commercial  potential  for  military,  university,  and  corporate  training  applications. 
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8.2  Conclusions 

We  believe  that  the  Phase  I  results  demonstrate  basic  feasibility  of  the  concept  prototype  for 
an  agent-enhanced  electronic  classroom  on  the  web.  The  study  was  specifically  structured  to  be 
narrow  in  scope,  but  of  sufficient  depth  to  set  the  foundation  and  to  provide  a  roadmap  for  a 
Phase  II  full-scope  development  and  validation  effort. 

Our  study  has  shown  that  the  agent-enhanced  approach  works  well  in  delivering  training  that 
is  personalized  to  each  individual  student  needs.  We  have  designed  and  demonstrated  a  Java- 
based  electronic  classroom  architecture  that  runs  in  the  web  browser  environment  and  is 
accessible  to  students  at  any  time,  from  any  location  in  the  world.  Our  effort  has  set  the 
foundation  for  a  web-based-training  course  delivery  mechanism  that  can  easily  be  customized  by 
instructors  for  a  variety  of  training  domains. 

In  summary,  we  believe  that  our  overall  Phase  I  effort  has  established  the  feasibility  of 
developing  a  full-scope  agent-enhanced  electronic  classroom  that  uses  the  Learn  Sesame 
personalization  engine  to  provide  the  highest  level  of  student  personalization  and  targeted 
training  on  the  internet. 

8.3  Recommendations 

On  the  basis  of  the  Phase  I  results,  we  reconunend  a  Phase  II  effort  to  develop,  validate,  and 
demonstrate  a  full-scope  agent-enhanced  electronic  classroom.  The  following  specific  objectives 
shown  in  Table  8.3-1  comprise  our  proposed  effort: 
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Table  8.3-1:  Features  of  Phase  I  and  Phase  II  Efforts 


Objective 


Focus 


Scope 


Approach 


Content  presentation 


Communication 


Knowledge  base 
maintenance 


Dynamic  FAQ 
maintenance 


Learning 


Example  scenario 


Establish  feasibility  of  electronic  classroom 


Content  presentation  and  multimedia 
delivery 

Communication 

Knowledge  base  maintenance  for  lesson 
Information 

Dynamic  FAQ  maintenance 
Example  electronic  classroom 


Single  classroom  with  limited  behavior 
learning  and  communication 


Concept  prototype  development  and 
demonstration 


Hypertext  for  text  presentation 
Java  interactive  applets 


Email 


Students  performance 


Exact  by  word  recognition 


Limited  learning  of  students  performance 


Aviation  weather  report  electronic 
classroom 


Develop  full-scope  prototype  electronic 
classroom 


Phase  I  features 

Recognition  of  learning  difficulties  and 
knowledge  deficiencies 

Learning  the  communication  style 
preferences  of  the  user 

Personalized  service  environment  for 
students 

Recognition  of  student’s  learning  style 


Multiple  classroom  with  full-scope  learning 
and  communication 


Prototype  enhancement,  demonstration 
and  evaluation 


Hypertext  and  interactive  applications 

Java  interactive  applets 

3-D  graphics 

Real-time  audio 

Video  clips  presentation 

Video  conferencing  _ 


Email 

Whiteboard,  Chat 

Collaboration  on  documents  between 
students  and  instructors 

Meetings  between  students  and  instructors 


Students  performance 
Students  learning  style 


Fuzzy  by  word  recognition 


Full-scale  learning  of  student’s 
performance 

Students  areas  of  mastery  or  difficulty 

Groups  of  students  areas  of  mastery  or 
difficulty 

Navigational  pattern  _ 


Enhancement  of  aviation  weather  report 
electronic  classroom 

MOUT  training  simulation 
Flight  plan  training 
Additional  university  classes 
Corporate  training 

UMass  WebCraft  electronic  classroom 
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10.  Appendix  A  •  Student  Evaluations 


1(None)  2  3  4  5  (A  lot) 

Aviation  Weather  Knowledge 


Figure  10.1*1:  How  much  did  you  know  about  Aviation  Weather 
before  the  MOS  93P  training  started? 


1  2  3  4  or  more 

Training  Setsions 


Figure  10.1-2:  How  many  training  session  did  you  have  on  the 
Internet  for  Aviation  Weather? 


1.0  1.5  20 

Training  Hours 


Figure  10.1-3:  How  much  total  time  did  you  spend  on  the  training? 
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Usefulness  of  the  Internet  Training 


Figure  10.1-4:  How  useful  did  you  find  the  Internet  training? 


6 
5 
4 
3 
2 
1 
0 

1  (No  Difficulty)  2  3  4  5  (A  Lot  of  Difficulty) 

Difficulty  of  the  Training 

Figure  10.1-5:  How  much  difficulty  did  you  have  using  the  Internet 

training? 


Helpfulness  of  the  Course 


Figure  10.1-6:  Compared  to  \ii^at  you  learned  about  Aviation 
Weather  during  classroom  training,  how  much  more  did  you  learn 
from  the  Internet  training? 
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Pn&vious  Internet  Experience 


Figure  10.1-7:  How  much  previous  hands-on  experience  did  you  have 

with  the  Internet? 


Previous  Computer  Experience 


Figure  10.1-8:  How  much  previous  hands-on  experience  did  you  have 

with  the  computers? 


Previous  Computer-based  Training 


Figure  10.1-9:  How  much  previous  hands-on  experience  did  you  have 
with  the  computer-based  training? 


A-3 


1  (No  Exptnenoo) 


2 


5  (Extensive  Expenence) 


Previous  E-malt  Experience 

Figure  10.1-10:  How  much  previous  hands-on  experience  did  you  have 

with  the  email? 


1  (Poor)  2  3  4  5  (Excellent) 


Usefulness  of  Sound 

Figure  10.1-11:  How  would  you  rate  the  sound  characteristics  of  the 
MOS  93P  Internet  training? 


1  (Poor)  2  3  4  5  (Excellent) 


Organization  of  Material 

Figure  10.1-12:  How  would  you  rate  the  organization  of  material? 
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CourM  Navigation 


Figure  10.1-13:  How  would  you  rate  the  course  navigation? 


1  (Poor)  2  3  4  5  (Excellent) 

Quality  of  Quizzes 


Figure  10.1-14:  How  would  you  rate  the  quality  of  quizzes? 


Amount  of  Information 

Figure  10.1-15:  How  would  you  rate  amount  of  information  provided 

in  each  weather  report? 
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Agent  Provided  Accurate  answers 


Figure  10.1-16:  The  agent  provided  me  with  accurate  answers  to  my 

questions. 


1  (Disagree)  2  3  4  5  (Agree) 


Agent  Provided  Helpful  guidance 

Figure  10.1-17:  The  agent  provided  helpful  guidance  based  on  my  quiz 

performance. 


6 

5 

4 

3 

2 

1 

0 


Figure  10.1-18:  The  agent  knew  when  I  was  ready  to  take  a  quiz. 


. 

1 

1  1 

1  (Disagree)  2  3  4  5  (Agree) 

Agent  knew  When  to  Offer  quiz 
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1  (Disagree)  2  3  4  5  (Agree) 

Agent  was  helpful 


Figure  10.1-19:  It  was  helpful  to  have  agent  always  available  to  me. 


6 
5 
4 
3 
2 
1 
0 

1  (Disagree)  2  3  4  5  (Agree) 

It  Is  easy  to  get  started 

Figure  10.1-20:  It  was  easy  to  get  started  using  the  eiectronic 

classroom. 


1  (Disagree)  2  3  4  5  (Agree) 

Easy  Interface 


Figure  10.1-21:  The  electronic  classroom  interface  was  easy  to  work 

with. 
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1  (Disagree)  2  3  4  5  (Agree) 


Lack  of  consistency 

Figure  10.1-22:  Lack  of  consistency  made  learning  the  made  learning 

the  material  difficult. 


Scroling  hinders  concentration 


Figure  10.1-23:  Scrolling  in  the  content  screen  interfered  with  my 

ability  to  concentrate. 


A-8 


